Orchestrating Data Across Heterogeneous Memory Hierarchies

337 concepts206 papers2 proposals

As workloads in deep learning, graph analytics, and large-scale scientific computing scale exponentially, they inevitably collide with a physical limitation known as the "memory wall." While processor compute power has skyrocketed, the bandwidth and capacity of traditional, uniform memory systems (such as standard DDR DRAM) have failed to keep pace.

To bypass this bottleneck, modern hardware is shifting away from uniform memory structures toward heterogeneous memory hierarchies. By combining diverse memory technologies—such as ultra-fast on-chip SRAM, high-bandwidth on-package memory (HBM), high-capacity volatile DRAM, and high-density Non-Volatile Memory (NVM) or Flash—systems can theoretically achieve both the low latency of the fastest tiers and the massive capacity of the cheapest tiers.

However, hardware heterogeneity introduces a fundamental challenge: how does the system ensure that the right piece of data resides in the right memory tier at the right time? Solving this requires moving away from transparent, one-size-fits-all hardware caching toward intelligent, software-directed, and hardware-assisted data orchestration.

Matching Data Semantics to Physical Media

The core insight behind heterogeneous memory orchestration is that application data is not uniform. Different data structures within the same application exhibit radically different access patterns, mutability, and latency sensitivities. By aligning these data semantics with the specific physical strengths of each memory tier, systems can optimize performance and energy efficiency.

Consider large-scale graph processing and recommendation systems. These workloads typically involve a mix of massive, read-heavy structural topologies (like graph edges or embedding tables) and highly dynamic, frequently updated states (like vertex properties). In NGraph: Parallel Graph Processing in Hybrid Memory Systems, Wei Liu 0004 et al. demonstrate that mapping large, sequentially accessed graph structures to high-capacity NVM while keeping frequently updated, randomly accessed states in low-latency DRAM eliminates the severe write bottlenecks inherent to NVM technologies. This targeted placement allows the system to achieve nearly the performance of an all-DRAM configuration at a fraction of the hardware cost.

┌────────────────────────────────────────────────────────┐
│               Application Data Semantics               │
└───────────────────────────┬────────────────────────────┘
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
   【 Read-Heavy / Static 】    【 Random / Frequently Updated 】
   e.g., LLM Weights, Graphs   e.g., KV Caches, Vertex States
              │                           │
              ▼                           ▼
┌───────────────────────────┐┌───────────────────────────┐
│ High-Capacity Tier (NVM)  ││  Low-Latency Tier (DRAM)  │
└───────────────────────────┘└───────────────────────────┘

This strategy is equally vital for Large Language Models (LLMs). During LLM token generation, the model must constantly access static model weights alongside a rapidly changing key-value (KV) cache. To scale this efficiently, Sangwoo Hwang et al. propose a hybrid storage strategy in Dissecting and Re-architecting 3D NAND Flash PIM Arrays for Efficient Single-Batch Token Generation in LLMs. They relegate static model weights to high-density multi-level cells (MLC) within Flash memory, while routing the highly dynamic, write-intensive KV cache to high-speed, high-endurance single-level cells (SLC), matching the physical endurance of the hardware to the operational volatility of the data.

Even in highly resource-constrained environments like edge AI, treating memory as a unified hierarchy yields massive benefits. In TinyOps: ImageNet Scale Deep Learning on Microcontrollers, Jonathon Hare et al. break the binary separation between internal and external microcontroller memory. By orchestrating a "ping-pong" buffering scheme via Direct Memory Access (DMA), they stream model layers from high-capacity external memory into fast internal SRAM just in time for execution, allowing tiny edge devices to run high-accuracy models without suffering debilitating latency penalties.

The Granularity Dilemma and Asymmetric Management

If data placement matters so much, why not let the operating system handle it transparently via traditional demand paging? The issue lies in a mismatch of granularity.

Traditional operating systems manage memory in coarse, fixed-size blocks (typically 4KB pages). However, application data structures are composed of fine-grained objects. As Ahmad Hassan et al. observe in Software-managed energy-efficient hybrid DRAM/NVM main memory, page-level management often suffers from a lack of "data purity." A single 4KB page frequently contains a mixture of both frequently accessed ("hot") and rarely accessed ("cold") objects. This dilution makes it impossible for the OS to make clean migration decisions, leading to unnecessary data thrashing between memory tiers.

To resolve this, recent architectures embrace asymmetric granularity—decoupling the size of data retrievals from the size of data evictions. In A Tale of Two Paths: Toward a Hybrid Data Plane for Efficient Far-Memory Applications, Lei Chen et al. design a hybrid data plane that fetches data from remote or slow memory as fine-grained objects, which minimizes I/O amplification. Crucially, however, the system evicts data back to the slow tier as coarse-grained pages. This asymmetry eliminates the massive CPU and memory overhead that would otherwise be required to track and maintain eviction metadata for millions of individual fine-grained objects.

Automating Orchestration via Runtimes and Compilers

Forcing application developers to manually track data locality and orchestrate memory transfers introduces severe programming complexity, as noted by Matheus A. Cavalcante et al. in MemPool: A Shared-L1 Memory Many-Core Cluster with a Low-Latency Interconnect. To make heterogeneous memory practical, the burden of orchestration must be shifted to intelligent software runtimes and compilers.

As highlighted by Xian‐He Sun et al. in their Preface on memory-centric design, software-defined memory management layers can transparently intercept application allocations and steer data to the optimal tier based on static analysis or real-time profiling.

Managed language runtimes (like the Java Virtual Machine) are uniquely positioned to perform this automation because they already control object lifecycles. In Efficient Management for Hybrid Memory in Managed Language Runtime, Ting Cao 0003 et al. demonstrate that instead of inventing new hardware mechanisms, runtimes can co-opt existing managed features like garbage collection object promotion and Just-In-Time (JIT) compilation. By tracking object age and access frequency during routine garbage collection cycles, the runtime can transparently promote "hot" objects into fast memory tiers and relegate "cold" objects to slower, high-capacity tiers without requiring a single modification to the underlying application code.

┌────────────────────────────────────────────────────────┐
│               Managed Language Runtime                 │
│  ┌──────────────────────┐    ┌──────────────────────┐  │
│  │   JIT Compilation    │    │  Garbage Collection  │  │
│  │ (Identifies Hot Path)│    │  (Tracks Object Age) │  │
│  └──────────┬───────────┘    └──────────┬───────────┘  │
└─────────────┼───────────────────────────┼──────────────┘
              └─────────────┬─────────────┘
                            ▼
┌────────────────────────────────────────────────────────┐
│         Transparent Memory Orchestration Layer         │
│  ┌──────────────────────┐    ┌──────────────────────┐  │
│  │ Fast Volatile Tier   │    │ High-Capacity Tier   │  │
│  │     (Hot Objects)    │    │    (Cold Objects)    │  │
│  └──────────────────────┘    └──────────────────────┘  │
└────────────────────────────────────────────────────────┘

Hardware-Assisted Hybrid Access Paths

While software runtimes provide high-level intelligence, they still rely on underlying hardware mechanisms to execute data transfers efficiently. Traditional memory controllers offer a rigid choice: they either route data through hardware-managed caches (which excel at exploiting temporal reuse but fail on irregular streaming data) or rely on explicit DMA engines (which handle bulk transfers efficiently but incur high setup overheads for small requests).

Modern heterogeneous systems break this binary constraint by implementing hybrid hardware access paths. Zhiyu Chen demonstrates the power of this approach in Programmable FPGA-based Memory Controller, proposing a controller architecture that provides separate, concurrent hardware paths for cache-line transfers and bulk DMA transfers. By routing irregular, reusable lookups through the cache while simultaneously streaming one-time bulk data via DMA, the architecture prevents streaming data from polluting the cache, maximizing bandwidth utilization across disparate memory technologies.

Furthermore, hardware flexibility allows the memory hierarchy to dynamically reconfigure its role based on the workload's scale. As Lei Jiang et al. explore in Performance Characterization of Multi-threaded Graph Processing Applications on Intel Many-Integrated-Core Architecture, 3D-stacked high-bandwidth memory can be configured to act either as a transparent, hardware-managed cache or as an explicitly addressable, software-directed memory tier. This allows system designers to trade the simplicity of hardware automation for the fine-grained, deterministic control of software orchestration depending on the predictability of the application's access patterns.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 198 papers
  2. Version 1 · 2026-10-04 · 159 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_memory_architecture_a35640bc,
  title = {hybrid memory architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-memory-architecture/rdr_a35640bce8794b4094d0f60339b8dfad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_hybrid_memory_architecture_a35640bc_v2,
  title = {hybrid memory architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-memory-architecture/rdr_a35640bce8794b4094d0f60339b8dfad/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake