Parent Log: http://ci.aztec-labs.com/6afe0e3c63790b84 Command: 4174e5df391694d3 yarn-project/scripts/run_test.sh archiver/src/archiver/kv_archiver_store/kv_archiver_store.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/8d254a16364bbc8136cad6732cc978ea1a895cae Env: REF_NAME=gh-readonly-queue/next/pr-15073-7763eef5eb7f045b9184a0a4591b394296eb3aa8 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 15:14:26 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15073_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_afeabe6543d27e83_next 15:14:29 [15:14:29.211] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:30 [15:14:30.671] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:32 [15:14:32.151] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:33 [15:14:33.040] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: InitialBlockNumberNotSequentialError: Cannot insert new block 2 given previous block number in store is undefined 15:14:33 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:79:15 15:14:33 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:14:33 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:14:33 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:14:33 newBlockNumber: 2, 15:14:33 previousBlockNumber: undefined 15:14:33 } 15:14:33 [15:14:33.046] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:34 [15:14:34.094] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: BlockNumberNotSequentialError: Cannot insert new block 3 given previous block number in batch is 1 15:14:34 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:86:17 15:14:34 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:14:34 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:14:34 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 15:14:34 [15:14:34.103] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:35 [15:14:35.518] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:36 [15:14:36.737] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:38 [15:14:38.239] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: Error: Can only unwind blocks from the tip (requested 5 but current tip is 10) 15:14:38 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:123:15 15:14:38 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:14:38 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:14:38 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 15:14:38 [15:14:38.244] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:40 [15:14:40.041] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:41 [15:14:41.658] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:43 [15:14:43.392] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:44 [15:14:44.952] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:46 [15:14:46.390] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:47 [15:14:47.812] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:49 [15:14:49.287] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:50 [15:14:50.864] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:52 [15:14:52.942] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:57 [15:14:57.439] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:14:58 [15:14:58.814] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:01 [15:15:01.057] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:02 [15:15:02.959] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:04 [15:15:04.778] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:06 [15:15:06.294] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:07 [15:15:07.868] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:09 [15:15:09.386] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:10 [15:15:10.864] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:12 [15:15:12.333] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:15 [15:15:15.024] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:17 [15:15:17.852] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:20 [15:15:20.499] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:23 [15:15:23.250] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:25 [15:15:25.589] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:28 [15:15:28.469] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:31 [15:15:31.149] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:33 [15:15:33.874] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:36 [15:15:36.510] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:39 [15:15:39.274] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:41 [15:15:41.755] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:44 [15:15:44.362] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:47 [15:15:47.586] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:49 [15:15:49.031] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:50 [15:15:50.499] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:51 [15:15:51.871] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:53 [15:15:53.435] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:54 [15:15:54.996] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:56 [15:15:56.716] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:15:58 [15:15:58.484] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:00 [15:16:00.475] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:02 [15:16:02.360] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:04 [15:16:04.270] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:06 [15:16:06.101] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x1becfd6caff776a46ba21f0606b93a312247e55538dcc566118ee31691dbb03a for L2 block 1 has wrong index 10 (expected 0) 15:16:06 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 15:16:06 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:06 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:06 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:06 inboxMessage: { 15:16:06 index: 10n, 15:16:06 leaf: Fr<0x1becfd6caff776a46ba21f0606b93a312247e55538dcc566118ee31691dbb03a>, 15:16:06 l2BlockNumber: 1, 15:16:06 l1BlockNumber: 68n, 15:16:06 l1BlockHash: Buffer32<0x688a3b8d0ade4d2072ed971be3ddc784d5dcecec43b4fdb8e459c00f71583b22>, 15:16:06 rollingHash: Buffer16<0x7e0362f188c49dd125a54500f44decb7> 15:16:06 } 15:16:06 } 15:16:06 [15:16:06.106] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:08 [15:16:08.066] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 224 for incoming L1 to L2 message 0x2a8fa27cf10a841a1646205a6868df215bdf062b9baab0d7a19cd7f87e673121 at block 12 (expected value in range [176, 192)) 15:16:08 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 15:16:08 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:08 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:08 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:08 inboxMessage: { 15:16:08 index: 224n, 15:16:08 leaf: Fr<0x2a8fa27cf10a841a1646205a6868df215bdf062b9baab0d7a19cd7f87e673121>, 15:16:08 l2BlockNumber: 12, 15:16:08 l1BlockNumber: 86n, 15:16:08 l1BlockHash: Buffer32<0x4239f9832b0bf407ceece82c22f68041caad83cd08b5860d693cbd74fbaee828>, 15:16:08 rollingHash: Buffer16<0xdccc40a5d76a98a1432921a43471602f> 15:16:08 } 15:16:08 } 15:16:08 [15:16:08.071] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:10 [15:16:10.074] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x279363b188fe2d03a9ab8b660070a1696b1e7a8537bd7455fb72a0eea8f0d20f with index 16 (expected 0xf7aa5a3da9d9fdd7832ccae93e14e828 from previous hash 0x0b5d6ed906cc0900a2cb636c04e9c891 but got 0x3bc81c309a68c7865054ed5ac8c810e9) 15:16:10 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 15:16:10 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:10 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:10 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:10 inboxMessage: { 15:16:10 index: 16n, 15:16:10 leaf: Fr<0x279363b188fe2d03a9ab8b660070a1696b1e7a8537bd7455fb72a0eea8f0d20f>, 15:16:10 l2BlockNumber: 2, 15:16:10 l1BlockNumber: 8n, 15:16:10 l1BlockHash: Buffer32<0x2011fc63a05440f222f407bc2876c99f9bfd7493d35abf4577318318612ee41f>, 15:16:10 rollingHash: Buffer16<0x3bc81c309a68c7865054ed5ac8c810e9> 15:16:10 } 15:16:10 } 15:16:10 [15:16:10.077] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:11 [15:16:11.887] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x25b5f15d56f28f2aed2746eac86d4d471d394c5c86c8227e1369b060588b7f39 with index 32 (expected 0x7f6defa8c445064d37a834f1a60ea440 from previous hash 0x164e078594b76ed3c79662b15860ff1b but got 0x6f48ef8a98e07a45bb1f90f6d0d29ace) 15:16:11 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 15:16:11 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:11 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:11 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:11 inboxMessage: { 15:16:11 index: 32n, 15:16:11 leaf: Fr<0x25b5f15d56f28f2aed2746eac86d4d471d394c5c86c8227e1369b060588b7f39>, 15:16:11 l2BlockNumber: 3, 15:16:11 l1BlockNumber: 8n, 15:16:11 l1BlockHash: Buffer32<0x0e1b1781eac542b7e5565b4f6e0828e586ecf4afa0063e80260b4be51f01d1d5>, 15:16:11 rollingHash: Buffer16<0x6f48ef8a98e07a45bb1f90f6d0d29ace> 15:16:11 } 15:16:11 } 15:16:11 [15:16:11.897] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:13 [15:16:13.682] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 356 for incoming L1 to L2 message 0x00b8308fc278e0b2441a27adb6ea8748566974f179f55298b8dae916bb319927 at block 17 (expected value in range [256, 272)) 15:16:13 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 15:16:13 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:13 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:13 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:13 inboxMessage: { 15:16:13 index: 356n, 15:16:13 leaf: Fr<0x00b8308fc278e0b2441a27adb6ea8748566974f179f55298b8dae916bb319927>, 15:16:13 l2BlockNumber: 17, 15:16:13 l1BlockNumber: 86n, 15:16:13 l1BlockHash: Buffer32<0xaef99f7fe4ce578d687a916333dee98dcc73cb54d5f3bb08c8c6add4b3c8dfd5>, 15:16:13 rollingHash: Buffer16<0xd6f4f373b84170429c92189eeac0b5a2> 15:16:13 } 15:16:13 } 15:16:13 [15:16:13.685] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:15 [15:16:15.629] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x04afae7d7ee002422f9f3dfed30a4aefacee771b4e339df9a362f40bf27de040 for L2 block 15 has wrong index 225 (expected 224) 15:16:15 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 15:16:15 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:15 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:15 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:15 inboxMessage: { 15:16:15 index: 225n, 15:16:15 leaf: Fr<0x04afae7d7ee002422f9f3dfed30a4aefacee771b4e339df9a362f40bf27de040>, 15:16:15 l2BlockNumber: 15, 15:16:15 l1BlockNumber: 35n, 15:16:15 l1BlockHash: Buffer32<0xb59915766697e5ea4dee65a85d0eedd99441009c99b094ee25ca847113079806>, 15:16:15 rollingHash: Buffer16<0xf2db935118c8c7485b7f714bc033e061> 15:16:15 } 15:16:15 } 15:16:15 [15:16:15.633] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:17 [15:16:17.602] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 32 for incoming L1 to L2 message 0x0495a69f5bf4ae70f248ce55fa3388b64111b342fd853c2ebd28560e53fa30e0 at block 2 (expected value in range [16, 32)) 15:16:17 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 15:16:17 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 15:16:17 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 15:16:17 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 15:16:17 inboxMessage: { 15:16:17 index: 32n, 15:16:17 leaf: Fr<0x0495a69f5bf4ae70f248ce55fa3388b64111b342fd853c2ebd28560e53fa30e0>, 15:16:17 l2BlockNumber: 2, 15:16:17 l1BlockNumber: 71n, 15:16:17 l1BlockHash: Buffer32<0x37ffbb5edfce48253883a53727af4d748fb4e10941ef5e04644ff8aad777e871>, 15:16:17 rollingHash: Buffer16<0x7040f49f98c6f21b57e80720164697c5> 15:16:17 } 15:16:17 } 15:16:17 [15:16:17.606] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:19 [15:16:19.701] WARN: archiver:message_store Deleted 32 L1 to L2 messages from index 32 from the store 15:16:19 [15:16:19.756] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:21 [15:16:21.754] WARN: archiver:message_store Deleted 51 L1 to L2 messages from index 13 from the store 15:16:21 [15:16:21.775] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:23 [15:16:23.616] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:25 [15:16:25.597] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:27 [15:16:27.387] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:29 [15:16:29.385] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 15:16:31 [15:16:31.262] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16