Memory Mapping for Out of Core Data Processing

39 concepts24 papers2 proposals

Data scales in modern scientific computing and machine learning are growing exponentially. Whether dealing with terabytes of satellite imagery, internet-scale text corpora, or high-resolution spatial omics, researchers frequently encounter a hard physical boundary: the dataset is too large to fit into system RAM.

The traditional workaround is cumbersome. Developers write custom pipelines to explicitly partition data, read chunks sequentially from disk, process them, and clear the memory. This introduces massive input/output (I/O) overhead due to constant serialization, context switching, and explicit buffer management.

Out-of-core data processing solves this by leveraging memory mapping (mmap) and lazy loading. Instead of loading a file into RAM, the operating system is instructed to treat the disk storage as an extension of the application's virtual address space. This article explores the core intuition behind why this mechanism works, how it eliminates I/O bottlenecks, and how it scales across diverse domains.

The Core Intuition: Virtual Memory and Page Faults

To understand why memory mapping works, we must look at how an operating system manages memory. When an application requests data from a file via traditional system calls like read(), the OS copies data from the disk into a kernel-space buffer, and then copies it again into a user-space buffer allocated in RAM.

Memory mapping bypasses this double-copy penalty. It creates a direct mapping between a file on disk and the application's virtual address space.

Traditional I/O:
[Disk] ---> [Kernel Page Cache] ---> [User Space RAM Buffer] (Double Copy)

Memory-Mapped I/O:
[Disk] ------------> [Virtual Address Space / Kernel Cache] (Zero/Near-Zero Copy)

When a file is memory-mapped, no data is actually read into physical RAM initially. Instead, the OS sets up page table entries pointing to the file.

  1. The application attempts to access an element in an array as if it were entirely resident in memory.
  2. The CPU's Memory Management Unit (MMU) realizes that the corresponding page of virtual memory is not yet backed by physical RAM.
  3. This triggers a hardware interrupt known as a page fault.
  4. The OS kernel transparently intercepts this fault, reads the required page directly from disk into physical memory, updates the page table, and resumes execution.

This process enables lazy loading: data is only read from disk at the exact moment it is required for a computation. This concept is not entirely new; early foundational systems like the PIC System II, described by Benes L. Trus et al. in Digital Image Processing of Electron Micrographs: The PIC System II, relied on virtual memory management to manipulate high-resolution micrographs that far exceeded the physical hardware limits of 1990s workstations.

Achieving Zero-Copy Retrieval in Modern Data Libraries

In modern data science, memory mapping is paired with structured, cross-language data layouts to achieve near-zero-copy retrieval.

In natural language processing, internet-scale text corpora present a massive memory burden. To address this, Albert Villanova del Moral et al. designed the Hugging Face datasets library (Datasets: A Community Library for Natural Language Processing) around Apache Arrow. Arrow utilizes a columnar data framework that can be memory-mapped directly from disk. Because the data layout on disk matches the layout in memory, looking up an entry requires zero deserialization overhead. The application simply reads the disk-backed virtual memory pointer, allowing consumer-grade hardware to manipulate massive datasets.

This technique is equally powerful for building interactive downstream applications. For instance, Odunayo Ogundepo et al. developed Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face, which uses memory-mapped data tables to handle massive search result sets. Web interfaces can instantly seek, retrieve, and display specific subsets of a corpus without the overhead of loading the full text index into RAM.

Chunking and Spatial Layouts for Multi-Dimensional Data

Memory mapping works flawlessly when data access patterns are sequential and contiguous. However, scientific datasets are often multi-dimensional. If a 3D medical image or a geospatial grid is stored naively row-by-row, attempting to read a vertical slice will force the OS to jump across disparate locations on disk, triggering a catastrophic cascade of page faults known as page thrashing.

To prevent this, out-of-core frameworks combine memory mapping with chunked storage formats (such as Zarr or OME-NGFF). Chunking breaks a massive multi-dimensional array into smaller, self-contained, contiguous blocks on disk.

Naive Row-Major Layout (Reading a vertical column causes page thrashing):
[Row 1..................][Row 2..................][Row 3..................]

Chunked Layout (Reading a local sub-region pulls exactly one contiguous block):
[Block A (x1,y1)] [Block B (x2,y1)] [Block C (x1,y2)] [Block D (x2,y2)]

In the field of spatial omics, Giovanni Palla et al. introduced SpatialData: an open and universal data framework for spatial omics. By storing high-resolution images and single-molecule transcripts in chunked formats, the framework ensures that spatial queries only read the specific segments required for a given computation, freeing local environments from physical RAM constraints.

Similarly, for planetary-scale Earth observation data, Benedikt Blumenstiel et al. leverage chunked, compressed array formats in TerraMesh: A Planetary Mosaic of Multimodal Earth Observation Data. This allows parallel, lazy access across high-performance computing clusters without needing to stage multi-terabyte volumes in memory.

For interactive visualization and annotation, this localized, on-demand loading is critical. Matthias Arzt et al. implemented this in LABKIT: Labeling and Segmentation Toolkit for Big Image Data, decoupling the visualization of multi-terabyte microscopy volumes from system RAM by loading and processing pixels exclusively within the user's active viewport.

Optimizing High-Throughput Machine Learning Pipelines

Machine learning training loops present a unique challenge for data loading: they require randomized, sliding-window, or batched access patterns at high throughput. If the data loader stalls while waiting for disk I/O, the GPU sits idle.

Memory mapping optimizes this ingestion pipeline by bypassing standard system call overheads entirely. In global weather forecasting, Christian Schroeder de Witt et al. developed the PyRain pipeline within RainBench: Towards Data-Driven Global Precipitation Forecasting from Satellite Imagery. By mapping physical disk space directly to virtual memory, PyRain handles the randomized sliding windows common in meteorological modeling, eliminating I/O bottlenecks without requiring specialized supercomputing hardware.

In offline reinforcement learning, where agents must learn from massive historical replay buffers, data management can easily bottleneck training. Albert Bou et al. addressed this in TorchRL: A data-driven decision-making library for PyTorch by introducing memory-mapped tensors. Storing large-scale datasets in contiguous physical memory via memory-mapped files minimizes I/O overhead, facilitating rapid data loading and batched transformations.

Traditional ML Pipeline:
[Disk] -> [DB Query / File Read] -> [Deserialize] -> [Batch Creation] -> [GPU] (Slow)

Memory-Mapped Tensor Pipeline:
[Memory-Mapped Disk File] -> [Direct Virtual Memory Pointer] -> [Batch] -> [GPU] (Fast)

Furthermore, applying core database principles like lazy evaluation and cursoring can optimize standard enterprise machine learning. As shown by Zeeshan Ahmed et al. in Machine Learning at Microsoft with ML.NET, encapsulating data streams within a lazy "DataView" abstraction allows high-performance training on large-scale data without ever loading the entire dataset into RAM.

Domain-Specific Applications: From Brains to the Cosmos

The necessity of treating disk storage as virtual memory spans across nearly every data-intensive scientific discipline:

The Frontier: Hardware-Coordinated Persistent Memory

While memory mapping historically bridged the gap between slow spinning disks and fast volatile DRAM, the emergence of Non-Volatile Persistent Memory (NVRAM) architectures is blurring the line between storage and memory entirely.

Andrew Wood et al. explored this hardware-software convergence in Non-Volatile Memory Accelerated Geometric Multi-Scale Resolution Analysis. By providing native support for standard numerical arrays directly within persistent memory frameworks (via the PyMM library), they demonstrated that memory-intensive algorithms can scale to terabyte-scale datasets with minimal code refactoring.

Remarkably, this approach incurs only a marginal performance penalty (as low as 4%) compared to traditional, pure in-memory execution. By co-designing software abstractions alongside hardware-level virtual memory mapping, out-of-core data processing ensures that physical RAM capacity is no longer a bottleneck for large-scale scientific discovery.

Key Foundational Work

Digital Image Processing of Electron Micrographs: The PIC System II (1991) — contributed 1 concept elements to this direction.
Exploring Parallelism in Learning Belief Networks (1997) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_memory_mapped_data_access_2d718cc1,
  title = {memory-mapped data access},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-mapped-data-access/rdr_2d718cc178bb4fa297996fe562d7093c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_memory_mapped_data_access_2d718cc1_v1,
  title = {memory-mapped data access},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-mapped-data-access/rdr_2d718cc178bb4fa297996fe562d7093c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake