Optimizing Data Locality for Non-Uniform Memory Access

237 concepts146 papers

In the early days of parallel computing, software developers could comfortably treat memory as a uniform pool. A byte of data took the same amount of time to fetch regardless of which CPU core requested it. This mental model—known as Uniform Memory Access (UMA)—is simple and clean, but it is completely broken on modern high-performance hardware.

As we scale systems to handle massive workloads like deep learning inference, real-time stream processing, and large-scale graph analytics, we inevitably rely on multi-socket servers and many-core architectures. In these environments, memory is physically distributed across distinct domains or sockets. This is the world of Non-Uniform Memory Access (NUMA).

The Intuition of the NUMA Tax

Why does NUMA matter? Imagine a processor core needs to read a variable. If that variable resides in the memory bank directly attached to its own socket (local access), the transaction is fast and direct. But if the variable resides in a memory bank attached to a different physical socket (remote access), the request must travel across an inter-socket interconnect (such as Intel's UPI or AMD's Infinity Fabric).

This remote journey introduces a massive latency penalty and consumes precious interconnect bandwidth. When hundreds of cores blindly execute tasks without considering where their data lives, the interconnects become saturated, caches are constantly invalidated, and system performance plateaus or even degrades. Optimizing for memory locality means rewriting software to respect this physical topology, ensuring that compute tasks and the data they require are always co-located.

The Anatomy of the NUMA Bottleneck

To understand why naive scaling fails, consider how data flows in a multi-socket system. As Andreas Kipf et al. demonstrate in Analytics on Fast Data: Main-Memory Database Systems versus Modern Streaming Systems, systems that lack explicit NUMA awareness suffer severe latency spikes and throughput drops when scaling across multiple CPU sockets. The core culprit is the high cost of remote memory access. Similarly, Yang You et al. note in Asynchronous Parallel Greedy Coordinate Descent that optimization algorithms frequently fail to achieve linear speedups because cores competing for the same memory bus cause severe hardware-level contention.

The problem extends beyond standard memory to I/O devices. In multi-socket systems, a Network Interface Card (NIC) is physically wired to a specific socket. If a thread running on Socket 1 tries to process packets from a NIC attached to Socket 0, the data must cross the inter-socket bus—a phenomenon known as Non-Uniform I/O Access (NUIOA). To bypass this, Wolf Rödiger et al. in High-Speed Query Processing over High-Speed Networks design their system to explicitly pin network processing threads and memory buffers to the specific CPU socket directly connected to the NIC, eliminating internal bus contention entirely.

Co-Locating Compute and Data via Deterministic Placement

The most direct way to eliminate the NUMA tax is to ensure that a thread and the data it operates on are pinned to the same physical domain. Instead of letting the operating system dynamically migrate threads or allocate memory pages arbitrarily, modern high-performance runtimes enforce deterministic placement.

For instance, in graph processing—where memory access patterns are notoriously irregular—Moritz Kaufmann et al. demonstrate in Parallel Array-Based Single- and Multi-Source Breadth First Searches on Large Dense Graphs that binding threads to specific NUMA nodes and ensuring their corresponding data pages are allocated on those exact same nodes neutralizes remote access penalties, enabling near-linear performance scaling. This strategy is echoed by Hanjiang Wu et al. in Enhancing Scalability and Performance in Influence Maximization with Optimized Parallel Processing, who use memory interleaving and explicit system calls to bind threads to specific sockets, minimizing high-latency inter-socket traffic.

In the context of deep learning inference, Jean-François David et al. show in Exploiting Processor Heterogeneity to Improve Throughput and Reduce Latency for Deep Neural Network Inference that blindly scaling the number of CPU cores leads to a performance plateau due to memory bus contention. By clustering processing cores according to their underlying NUMA nodes, they maintain strict data locality, keeping computational tasks close to the specific memory controllers and caches associated with those cores.

Mitigating Contention via Local Data Replication

Co-location works perfectly when data can be cleanly partitioned. But what happens when multiple threads across different sockets need to read the same global data structure? If all sockets target a single memory node, that node's memory controller becomes a massive bottleneck.

The solution is local replication: sacrificing a small amount of memory capacity to replicate read-only data structures across every NUMA domain. Yan Gu leverages this insight in Sage: Parallel Semi-Asymmetric Graph Algorithms for NVRAMs by replicating read-only graph structures across the local non-volatile memory of each CPU socket. This ensures that processing units can fetch graph data locally, achieving over a six-fold speedup by completely avoiding the interconnect bottleneck. Similarly, Erik Schultheis et al. show in Extreme Multicore Classification that replicating a feature matrix across distinct NUMA domains drastically reduces memory bus contention and latency compared to a single shared-memory configuration.

Hierarchical Scheduling and Locality-Aware Work-Stealing

Static partitioning and replication work well for predictable workloads, but real-world applications require dynamic load balancing. Traditional parallel runtimes use global task queues or unrestricted work-stealing, where an idle core can steal a task from any other core. On a NUMA system, if a core on Socket 1 steals a task whose data resides on Socket 0, the cost of moving that task's data across the interconnect can easily exceed the actual execution time.

To solve this, modern schedulers adopt a hierarchical, locality-aware approach. Jiawen Sun et al. introduce this mechanism in GraphGrind: addressing load imbalance of graph partitioning, modifying parallel runtime work-stealing behaviors to map loop iterations directly to specific NUMA domains, preserving data locality while maintaining global load balance.

Furthermore, Song Wu et al. demonstrate in A Real-Time Scheduling Framework Based on Multi-core Dynamic Partitioning in Virtualized Environment that restricting global scheduling queues to a subset of cores within a single NUMA node preserves cache locality and eliminates the severe cache-miss penalties that occur when virtual CPUs migrate across physical sockets. When fine-grained parallelism is required, Wenyi Wang et al. show in Optimizing Fine-Grained Parallelism Through Dynamic Load Balancing on Multi-Socket Many-Core Systems that prioritizing task migration strictly within the same NUMA node prevents high cross-socket communication costs, unlocking massive speedups for fine-grained parallel tasks.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 140 papers
  2. Version 1 · 2026-10-04 · 149 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_memory_locality_optimization_747ee221,
  title = {memory locality optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-locality-optimization/rdr_747ee221d1e640a0887dd2656539537e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_memory_locality_optimization_747ee221_v2,
  title = {memory locality optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-locality-optimization/rdr_747ee221d1e640a0887dd2656539537e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake