Parent Log: http://ci.aztec-labs.com/59a7f67799d27b25 Command: 7a41fc0e1576329f yarn-project/scripts/run_test.sh archiver/src/archiver/kv_archiver_store/kv_archiver_store.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/142e6cd5fbf7371f649078f9d6d08828585c664d Env: REF_NAME=gh-readonly-queue/next/pr-14610-016def8adbfe93f34b41a41f439b146b5318c895 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 24 12:56:44 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14610_amd64_x3-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_afeabe6543d27e83_next 12:56:47 [12:56:47.507] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:48 [12:56:48.913] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:50 [12:56:50.127] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:50 [12:56:50.925] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: InitialBlockNumberNotSequentialError: Cannot insert new block 2 given previous block number in store is undefined 12:56:50 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:89:15 12:56:50 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:56:50 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:56:50 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:56:50 newBlockNumber: 2, 12:56:50 previousBlockNumber: undefined 12:56:50 } 12:56:50 [12:56:50.928] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:51 [12:56:51.830] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: BlockNumberNotSequentialError: Cannot insert new block 3 given previous block number in batch is 1 12:56:51 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:96:17 12:56:51 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:56:51 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:56:51 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 12:56:51 [12:56:51.842] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:53 [12:56:53.376] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:54 [12:56:54.594] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:55 [12:56:55.822] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: Error: Can only unwind blocks from the tip (requested 5 but current tip is 10) 12:56:55 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/block_store.ts:141:15 12:56:55 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:56:55 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:56:55 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) 12:56:55 [12:56:55.828] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:57 [12:56:57.129] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:58 [12:56:58.362] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:56:59 [12:56:59.999] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:01 [12:57:01.631] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:03 [12:57:03.049] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:04 [12:57:04.593] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:06 [12:57:06.860] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:08 [12:57:08.949] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:11 [12:57:11.288] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:14 [12:57:14.799] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:16 [12:57:16.469] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:18 [12:57:18.736] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:20 [12:57:20.515] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:22 [12:57:22.275] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:23 [12:57:23.773] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:25 [12:57:25.485] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:26 [12:57:26.917] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:28 [12:57:28.393] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:29 [12:57:29.991] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:32 [12:57:32.649] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:35 [12:57:35.343] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:38 [12:57:38.085] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:40 [12:57:40.772] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:43 [12:57:43.076] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:45 [12:57:45.607] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:48 [12:57:48.039] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:50 [12:57:50.864] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:53 [12:57:53.558] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:56 [12:57:56.306] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:57:59 [12:57:59.085] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:01 [12:58:01.586] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:04 [12:58:04.734] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:06 [12:58:06.248] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:07 [12:58:07.817] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:09 [12:58:09.371] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:10 [12:58:10.835] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:12 [12:58:12.510] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:14 [12:58:14.168] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:15 [12:58:15.676] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:17 [12:58:17.370] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:19 [12:58:19.199] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:21 [12:58:21.040] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:22 [12:58:22.781] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x22095d1b16b2f3f484d6f1d35e114e951e4fb39d5caa9b0763d4e558221d5153 for L2 block 1 has wrong index 10 (expected 0) 12:58:22 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 12:58:22 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:22 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:22 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:22 inboxMessage: { 12:58:22 index: 10n, 12:58:22 leaf: Fr<0x22095d1b16b2f3f484d6f1d35e114e951e4fb39d5caa9b0763d4e558221d5153>, 12:58:22 l2BlockNumber: 1, 12:58:22 l1BlockNumber: 64n, 12:58:22 l1BlockHash: Buffer32<0x6565b87f73dc6b3576078336acfb996c6b7e7bc788b8e91365946158bc73d0d5>, 12:58:22 rollingHash: Buffer16<0xc2343c0931cffc2bc9457371cd63360e> 12:58:22 } 12:58:22 } 12:58:22 [12:58:22.785] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:24 [12:58:24.475] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 224 for incoming L1 to L2 message 0x14be9c9a384ba8d7df242f4cf7f473e0a994180ea933855f149d59db343b64dc at block 12 (expected value in range [176, 192)) 12:58:24 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 12:58:24 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:24 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:24 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:24 inboxMessage: { 12:58:24 index: 224n, 12:58:24 leaf: Fr<0x14be9c9a384ba8d7df242f4cf7f473e0a994180ea933855f149d59db343b64dc>, 12:58:24 l2BlockNumber: 12, 12:58:24 l1BlockNumber: 39n, 12:58:24 l1BlockHash: Buffer32<0xf3ce7ef1587709173adf6b8e2141da5fe815bab9aa8c24d26bf414834666ec73>, 12:58:24 rollingHash: Buffer16<0x7d665f843688a0f9cfa1bc9bafc1efce> 12:58:24 } 12:58:24 } 12:58:24 [12:58:24.478] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:26 [12:58:26.205] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x0d858ad1a9946f5b987c5705915e1cf2e703229e1f600ec8317ec254b689ff28 with index 16 (expected 0xbce9c081c5c4aa2a86b0ced07832b221 from previous hash 0x1ed05f5ffdc3e866c2ea9df4ad00e3a4 but got 0xc68573bdc070cca2dcfd1ba1fc94f026) 12:58:26 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 12:58:26 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:26 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:26 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:26 inboxMessage: { 12:58:26 index: 16n, 12:58:26 leaf: Fr<0x0d858ad1a9946f5b987c5705915e1cf2e703229e1f600ec8317ec254b689ff28>, 12:58:26 l2BlockNumber: 2, 12:58:26 l1BlockNumber: 91n, 12:58:26 l1BlockHash: Buffer32<0x4eb86ba525b226d39367068af6415f8853bd6fa235a98326d12dcf0eb7dc18db>, 12:58:26 rollingHash: Buffer16<0xc68573bdc070cca2dcfd1ba1fc94f026> 12:58:26 } 12:58:26 } 12:58:26 [12:58:26.209] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:28 [12:58:28.021] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid rolling hash for incoming L1 to L2 message 0x13d04acaa6c5f94b004ecf2e6b903d70cf52e6e94a5cc4d696a0f46ae7548163 with index 32 (expected 0x9ee8ca608f335f0f41661f8527af6e47 from previous hash 0xa76c465ba234e9f516160d0f6475e9ba but got 0x137d5c463bd93607ef3abaebfbacc6bb) 12:58:28 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:108:17 12:58:28 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:28 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:28 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:28 inboxMessage: { 12:58:28 index: 32n, 12:58:28 leaf: Fr<0x13d04acaa6c5f94b004ecf2e6b903d70cf52e6e94a5cc4d696a0f46ae7548163>, 12:58:28 l2BlockNumber: 3, 12:58:28 l1BlockNumber: 48n, 12:58:28 l1BlockHash: Buffer32<0xee681b8800ab126f944bf5b58d52a1b5dc8bfd228292e710059d925f9071bd37>, 12:58:28 rollingHash: Buffer16<0x137d5c463bd93607ef3abaebfbacc6bb> 12:58:28 } 12:58:28 } 12:58:28 [12:58:28.024] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:29 [12:58:29.864] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 356 for incoming L1 to L2 message 0x00910f877daf572d8db6d91450685a918990bc4e1261cc9aff83937f6f5aa651 at block 17 (expected value in range [256, 272)) 12:58:29 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 12:58:29 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:29 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:29 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:29 inboxMessage: { 12:58:29 index: 356n, 12:58:29 leaf: Fr<0x00910f877daf572d8db6d91450685a918990bc4e1261cc9aff83937f6f5aa651>, 12:58:29 l2BlockNumber: 17, 12:58:29 l1BlockNumber: 76n, 12:58:29 l1BlockHash: Buffer32<0xa5d18a6dae25d0dac2776e1c52f6e6523423ff8c0a9a66ba78807f84524905ed>, 12:58:29 rollingHash: Buffer16<0x7f1c388b284868d6a6ca21bac5a32589> 12:58:29 } 12:58:29 } 12:58:29 [12:58:29.868] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:31 [12:58:31.600] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Message 0x1992ec9775638b894d34a8f589fa2baa238106fb4ae6209c7cef9793a935ea84 for L2 block 15 has wrong index 225 (expected 224) 12:58:31 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:144:17 12:58:31 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:31 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:31 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:31 inboxMessage: { 12:58:31 index: 225n, 12:58:31 leaf: Fr<0x1992ec9775638b894d34a8f589fa2baa238106fb4ae6209c7cef9793a935ea84>, 12:58:31 l2BlockNumber: 15, 12:58:31 l1BlockNumber: 24n, 12:58:31 l1BlockHash: Buffer32<0xb117a87867690d1a704430dbf06ccfe674ccfef7c8ca20e8a2c11be611834de2>, 12:58:31 rollingHash: Buffer16<0xf56bbdeee525921bd2900ddf1b3f3bda> 12:58:31 } 12:58:31 } 12:58:31 [12:58:31.604] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:33 [12:58:33.430] ERROR: kv-store:lmdb-v2:archiver_test Failed to commit transaction: MessageStoreError: Invalid index 32 for incoming L1 to L2 message 0x2f28576f5ce7fe94d410a9a696ec852466bfe73974f30273a974a681c12b872e at block 2 (expected value in range [16, 32)) 12:58:33 at /home/aztec-dev/aztec-packages/yarn-project/archiver/src/archiver/kv_archiver_store/message_store.ts:119:17 12:58:33 at /home/aztec-dev/aztec-packages/yarn-project/kv-store/dest/lmdb-v2/store.js:111:29 12:58:33 at /home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/serial_queue.js:58:33 12:58:33 at FifoMemoryQueue.process (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/queue/base_memory_queue.js:110:17) { 12:58:33 inboxMessage: { 12:58:33 index: 32n, 12:58:33 leaf: Fr<0x2f28576f5ce7fe94d410a9a696ec852466bfe73974f30273a974a681c12b872e>, 12:58:33 l2BlockNumber: 2, 12:58:33 l1BlockNumber: 18n, 12:58:33 l1BlockHash: Buffer32<0x38b4f03a9d140581c2ca4a927217a084f75e394e7a98bc38b9cddc5beba05af3>, 12:58:33 rollingHash: Buffer16<0x9461287749fc90815d97d2796694004f> 12:58:33 } 12:58:33 } 12:58:33 [12:58:33.434] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:35 [12:58:35.460] WARN: archiver:message_store Deleted 32 L1 to L2 messages from index 32 from the store 12:58:35 [12:58:35.499] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:37 [12:58:37.225] WARN: archiver:message_store Deleted 51 L1 to L2 messages from index 13 from the store 12:58:37 [12:58:37.249] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:39 [12:58:39.166] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:40 [12:58:40.830] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:42 [12:58:42.564] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:44 [12:58:44.305] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:45 [12:58:45.995] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:47 [12:58:47.867] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:49 [12:58:49.693] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:51 [12:58:51.512] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:53 [12:58:53.269] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:55 [12:58:55.092] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:56 [12:58:56.935] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:58:58 [12:58:58.699] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:59:00 [12:59:00.406] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:59:02 [12:59:02.146] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16 12:59:03 [12:59:03.913] INFO: kv-store:lmdb-v2:archiver_test Starting data store with maxReaders 16