Parent Log: http://ci.aztec-labs.com/5dcd1b40cb4c7fca Command: 7a41fc0e1576329f yarn-project/scripts/run_test.sh archiver/src/archiver/kv_archiver_store/kv_archiver_store.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/142e6cd5fbf7371f649078f9d6d08828585c664d Env: REF_NAME=gh-readonly-queue/next/pr-14610-016def8adbfe93f34b41a41f439b146b5318c895 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 24 12:56:49 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14610_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_afeabe6543d27e83_next 12:56:52 [12:56:52.509] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:53 [12:56:53.929] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:55 [12:56:55.623] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:56 [12:56:56.698] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: InitialBlockNumberNotSequentialError: Cannot insert new block 2 given previous block number in store is undefined 12:56:56 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:89:15 12:56:56 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:56:56 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:56:56 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:56:56 newBlockNumber: 2, 12:56:56 previousBlockNumber: undefined 12:56:56 } 12:56:56 [12:56:56.702] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:57 [12:56:57.740] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: BlockNumberNotSequentialError: Cannot insert new block 3 given previous block number in batch is 1 12:56:57 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:96:17 12:56:57 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:56:57 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:56:57 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 12:56:57 [12:56:57.744] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:59 [12:56:59.239] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:00 [12:57:00.104] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:01 [12:57:01.377] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: Error: Can only unwind blocks from the tip (requested 5 but current tip is 10) 12:57:01 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:141:15 12:57:01 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:57:01 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:57:01 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 12:57:01 [12:57:01.380] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:02 [12:57:02.888] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:04 [12:57:04.282] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:06 [12:57:06.115] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:07 [12:57:07.727] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:09 [12:57:09.329] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:10 [12:57:10.860] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:13 [12:57:13.647] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:16 [12:57:16.573] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:18 [12:57:18.163] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:21 [12:57:21.656] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:23 [12:57:23.016] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:24 [12:57:24.951] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:26 [12:57:26.894] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:28 [12:57:28.483] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:29 [12:57:29.740] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:31 [12:57:31.230] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:32 [12:57:32.693] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:34 [12:57:34.381] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:35 [12:57:35.972] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:38 [12:57:38.671] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:41 [12:57:41.608] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:44 [12:57:44.242] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:46 [12:57:46.789] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:49 [12:57:49.296] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:52 [12:57:52.021] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:54 [12:57:54.887] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:57 [12:57:57.629] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:00 [12:58:00.714] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:03 [12:58:03.480] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:06 [12:58:06.463] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:08 [12:58:08.842] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:12 [12:58:12.114] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:13 [12:58:13.714] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:15 [12:58:15.319] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:16 [12:58:16.917] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:18 [12:58:18.577] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:20 [12:58:20.155] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:21 [12:58:21.755] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:23 [12:58:23.490] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:25 [12:58:25.314] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:27 [12:58:27.136] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:28 [12:58:28.992] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:30 [12:58:30.676] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x1fffe12a8ec668bd04d8f836117cd110f714fbf4859177e8d6580e7d60c91fde for L2 block 1 has wrong index 10 (expected 0) 12:58:30 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 12:58:30 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:30 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:30 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:30 inboxMessage: { 12:58:30 index: 10n, 12:58:30 leaf: Fr<0x1fffe12a8ec668bd04d8f836117cd110f714fbf4859177e8d6580e7d60c91fde>, 12:58:30 l2BlockNumber: 1, 12:58:30 l1BlockNumber: 48n, 12:58:30 l1BlockHash: Buffer32<0xc7713fa470f7021ae0609802a4cee29ddc89a336983dd563a83bae6afba95733>, 12:58:30 rollingHash: Buffer16<0x917993f0eb7ef8814d3193dfce91bee4> 12:58:30 } 12:58:30 } 12:58:30 [12:58:30.684] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:32 [12:58:32.432] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 224 for incoming L1 to L2 message 0x195917ae391971430c6ea1fbfe2999ad6ed05ad6c1de1629f469f3106ad2e51c at block 12 (expected value in range [176, 192)) 12:58:32 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 12:58:32 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:32 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:32 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:32 inboxMessage: { 12:58:32 index: 224n, 12:58:32 leaf: Fr<0x195917ae391971430c6ea1fbfe2999ad6ed05ad6c1de1629f469f3106ad2e51c>, 12:58:32 l2BlockNumber: 12, 12:58:32 l1BlockNumber: 73n, 12:58:32 l1BlockHash: Buffer32<0x62389389d4a46d3c5d91adbb8dc039631ea0e388b60f0fe20e6a6c94babbec86>, 12:58:32 rollingHash: Buffer16<0xc07a9745a566f2c4b246b432eb7bd226> 12:58:32 } 12:58:32 } 12:58:32 [12:58:32.440] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:34 [12:58:34.291] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x20863b6f68aa9acbebf7811fb3396bbd2b1747c0ad7d3f0ad651a401a9572d7e with index 16 (expected 0xf7f3428acf9098f3051bb1d4134139d2 from previous hash 0x8c15383e18dfb69fabdc537b04765321 but got 0xc3ce59384e3283467b7272df74086e70) 12:58:34 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 12:58:34 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:34 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:34 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:34 inboxMessage: { 12:58:34 index: 16n, 12:58:34 leaf: Fr<0x20863b6f68aa9acbebf7811fb3396bbd2b1747c0ad7d3f0ad651a401a9572d7e>, 12:58:34 l2BlockNumber: 2, 12:58:34 l1BlockNumber: 98n, 12:58:34 l1BlockHash: Buffer32<0x34bdc945def0946a1aed867f8272a561656c347e8cd2351181f0f60b72f51c99>, 12:58:34 rollingHash: Buffer16<0xc3ce59384e3283467b7272df74086e70> 12:58:34 } 12:58:34 } 12:58:34 [12:58:34.297] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:36 [12:58:36.040] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x15bf6570c96769295e88280cea8bd99a69fbb7bd6cd91b3252921265178df043 with index 32 (expected 0x26d8f33e4f54f607f155552048491fe5 from previous hash 0x602580a88dad209808934715f6441133 but got 0x28a2686c1d43e66c927b0e3ba65c009b) 12:58:36 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 12:58:36 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:36 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:36 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:36 inboxMessage: { 12:58:36 index: 32n, 12:58:36 leaf: Fr<0x15bf6570c96769295e88280cea8bd99a69fbb7bd6cd91b3252921265178df043>, 12:58:36 l2BlockNumber: 3, 12:58:36 l1BlockNumber: 40n, 12:58:36 l1BlockHash: Buffer32<0x70134f45bbe4f0e8731b12ec45c2e7da849fc5fd7a0505b1c32f96baeee3aa77>, 12:58:36 rollingHash: Buffer16<0x28a2686c1d43e66c927b0e3ba65c009b> 12:58:36 } 12:58:36 } 12:58:36 [12:58:36.048] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:37 [12:58:37.899] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 356 for incoming L1 to L2 message 0x2b715b51cde264c9e04456caa487d23dfe5055569c57b92a811768ba64c7394d at block 17 (expected value in range [256, 272)) 12:58:37 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 12:58:37 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:37 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:37 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:37 inboxMessage: { 12:58:37 index: 356n, 12:58:37 leaf: Fr<0x2b715b51cde264c9e04456caa487d23dfe5055569c57b92a811768ba64c7394d>, 12:58:37 l2BlockNumber: 17, 12:58:37 l1BlockNumber: 12n, 12:58:37 l1BlockHash: Buffer32<0x28742aba0e20a7aef75b66f3a655671faf5b4be62ab3dd2a7031c5ae4171e4aa>, 12:58:37 rollingHash: Buffer16<0x3bd8cadda1c1ec3c0dde34cb28a5c398> 12:58:37 } 12:58:37 } 12:58:37 [12:58:37.902] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:39 [12:58:39.458] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x067bb22dbf45ec9c56069649867831b400dada72c960e0eb121dd929057537c0 for L2 block 15 has wrong index 225 (expected 224) 12:58:39 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 12:58:39 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:39 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:39 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:39 inboxMessage: { 12:58:39 index: 225n, 12:58:39 leaf: Fr<0x067bb22dbf45ec9c56069649867831b400dada72c960e0eb121dd929057537c0>, 12:58:39 l2BlockNumber: 15, 12:58:39 l1BlockNumber: 43n, 12:58:39 l1BlockHash: Buffer32<0x37ca13766085f506d243631298919a5c87da320a134f2dc36b95171cac357028>, 12:58:39 rollingHash: Buffer16<0x4be37406e385ba0bc2768f312fa614c5> 12:58:39 } 12:58:39 } 12:58:39 [12:58:39.461] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16