ACI ·

Parent Log: http://ci.aztec-labs.com/0a191b42feca0cf7 Command: 200d11860e478e79 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/broker_prover_facade.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/00cd67e98e76d3202a0219baf301d59ef4211872 Env: REF_NAME=gh-readonly-queue/next/pr-15040-7339c264ad706420ee9babb0dd1b6ed793e97402 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Fri Jun 13 15:21:11 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15040_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b85794959204d424_next 15:21:16 [15:21:16.181] INFO: prover-client:proving-broker Proving Broker started 15:21:16 [15:21:16.194] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:16 [15:21:16.293] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 15:21:16 [15:21:16.296] INFO: prover-client:proving-agent:job-controller-8428d84b Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:16 [15:21:16.300] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 15:21:16 [15:21:16.301] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:17 [15:21:17.199] INFO: prover-client:proving-broker Proving Broker started 15:21:17 [15:21:17.208] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:17 [15:21:17.301] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 15:21:17 [15:21:17.303] INFO: prover-client:proving-agent:job-controller-dadc6299 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:17 [15:21:17.308] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 15:21:17 [15:21:17.308] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:18 [15:21:18.204] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:19 [15:21:19.534] INFO: prover-client:proving-broker Proving Broker started 15:21:19 [15:21:19.539] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:19 [15:21:19.637] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 15:21:19 [15:21:19.638] INFO: prover-client:proving-agent:job-controller-bd3b02ac Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:19 [15:21:19.652] ERROR: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY failed err=TEST ERROR retry=false: Error: TEST ERROR 15:21:19 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.test.ts:124:26) 15:21:19 at runNextTicks (node:internal/process/task_queues:65:5) 15:21:19 at listOnTimeout (node:internal/timers:549:9) 15:21:19 at processTimers (node:internal/timers:523:7) 15:21:19 [15:21:19.653] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=TEST ERROR {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:20 [15:21:20.537] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 15:21:20 [15:21:20.538] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 15:21:20 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 15:21:20 provingJobType: 'BASE_PARITY' 15:21:20 } 15:21:20 [15:21:20.541] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:20 [15:21:20.543] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 15:21:21 [15:21:21.541] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 15:21:21 [15:21:21.542] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 15:21:21 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 15:21:21 provingJobType: 'BASE_PARITY' 15:21:21 } 15:21:21 [15:21:21.543] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 15:21:21 [15:21:21.547] INFO: prover-client:proving-broker Proving Broker started 15:21:21 [15:21:21.548] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.648] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 15:21:21 [15:21:21.649] INFO: prover-client:proving-agent:job-controller-f4b1bbb3 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.650] INFO: prover-client:proving-broker Cancelling job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.650] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.750] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.851] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:21 [15:21:21.953] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.054] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.155] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.256] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.357] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.459] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.548] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'Aborted' 15:21:22 [15:21:22.548] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Aborted' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 15:21:22 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 15:21:22 provingJobType: 'BASE_PARITY' 15:21:22 } 15:21:22 [15:21:22.549] WARN: prover-client:proving-agent:job-controller-f4b1bbb3 Aborted job controller for jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.550] INFO: prover-client:proving-broker Proving Broker started 15:21:22 [15:21:22.551] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 15:21:22 [15:21:22.552] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 15:21:22 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 15:21:22 provingJobType: 'BASE_PARITY' 15:21:22 } 15:21:22 [15:21:22.553] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 15:21:22 [15:21:22.554] INFO: prover-client:proving-broker Proving Broker started 15:21:22 [15:21:22.655] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 15:21:22 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 15:21:22 provingJobType: 'BASE_PARITY' 15:21:22 } 15:21:22 [15:21:22.666] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 15:21:22 PASS src/proving_broker/broker_prover_facade.test.ts (9.984 s) 15:21:22 BrokerCircuitProverFacade 15:21:22 ✓ sends jobs to the broker (1020 ms) 15:21:22 ✓ handles multiple calls for the same job (2333 ms) 15:21:22 ✓ handles proof errors (2014 ms) 15:21:22 ✓ handles aborts (1003 ms) 15:21:22 ✓ rejects jobs when the facade is stopped (3 ms) 15:21:22 ✓ handles stopping while sending a proof to the broker (113 ms) 15:21:22 15:21:22 Test Suites: 1 passed, 1 total 15:21:22 Tests: 6 passed, 6 total 15:21:22 Snapshots: 0 total 15:21:22 Time: 10.071 s 15:21:22 Ran all test suites matching /prover-client\/src\/proving_broker\/broker_prover_facade.test.ts/i. 15:21:22 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?