Shared Virtual Addressing for Zero Copy Heterogeneous Acceleration

58 concepts44 papers1 proposal

In traditional computer architectures, processors and accelerators live in isolated worlds. A host CPU manages general system logic, while accelerators—such as GPUs, FPGAs, or specialized neural engines—handle parallel arithmetic.

This division comes with a heavy tax: memory isolation. Because these processing units maintain separate memory spaces, transferring data between them requires explicit serialization, pointer translation, and Direct Memory Access (DMA) copies over a peripheral bus like PCIe. This "copy tax" introduces massive latency, wastes energy, and forces developers to write complex code to manually manage data synchronization.

To break down this wall, modern research focuses on coherent shared memory architectures and Shared Virtual Addressing (SVA). By unifying the virtual address space and tightly coupling the physical memory hierarchy, CPUs and accelerators can share pointers directly, enabling true zero-copy data exchange.


The Pointer Problem and Shared Virtual Addressing

To understand why unified memory is transformative, consider how a computer handles complex data structures like graphs, trees, or linked lists. These structures rely on pointers—memory addresses that point to other pieces of data.

In a traditional split-memory architecture, if a CPU passes a pointer to an accelerator, that pointer is useless because the accelerator's memory controller maps addresses differently. The data must be flattened (serialized), copied over, and reconstructed with new pointers (pointer swizzling) before the accelerator can use it.

Shared Virtual Addressing solves this by ensuring that a virtual address points to the exact same physical data, whether accessed by the CPU or an accelerator. Early work by Gelado et al. demonstrated that centralizing memory coherence management on the host CPU allows complex pointer-rich data structures to be passed directly to accelerator kernels, eliminating manual pointer translation entirely.

Traditional Architecture:
[ CPU Virtual Space ] --(Serialize & Copy)--> [ Bus ] --(Reconstruct)--> [ Accelerator Virtual Space ]

Shared Virtual Addressing:
[ CPU ] ---------\
                  +---> [ Unified Virtual Address Space ] ---> [ Shared Physical Memory ]
[ Accelerator ] --/

In modern hardware, this unification is enforced by an Input-Output Memory Management Unit (IOMMU) or by sharing a hardware Memory Management Unit (MMU). For example, Kurth et al. designed a hybrid IOMMU that allows many-core accelerator clusters to seamlessly share a virtual address space with a host processor. For tightly integrated cores, Perotti showed that sharing a single MMU between a scalar core and a vector engine provides a highly area-efficient way to support full virtual memory without duplicating hardware logic.

However, translating addresses through an IOMMU introduces latency. Koenig et al. evaluated this performance penalty, noting that while SVA vastly simplifies programming, address translation can introduce up to a 17.6% performance overhead. Crucially, the authors demonstrated that this penalty can be mitigated to under 1% by integrating a shared last-level cache (LLC) to keep translation structures and data localized.


Tight Coupling at the Hardware Level

Aligning virtual addresses is only half the solution; the physical hardware must also be tightly coupled to prevent data movement bottlenecks. This coupling happens at two scales: on-chip shared caches for edge devices, and cache-coherent interconnects for high-performance systems.

On-Chip Shared Scratchpads

In ultra-low-power edge devices and Systems-on-Chip (SoCs), accelerators are placed directly alongside CPU cores to share primary cache tiers. Conti et al. demonstrated that connecting programmable cores and specialized accelerators to a common Level 1 data memory eliminates the power-intensive overhead of moving data over a traditional system bus.

This architectural pattern is highly effective for mathematical acceleration. Tortorella et al. integrated matrix-multiplication engines that share a multi-banked L1 memory with RISC-V cores, allowing the host to offload heavy linear algebra tasks with minimal latency. By allowing the accelerator to process data in place within the primary memory tier, systems can bypass the limitations of traditional system buses entirely (Garofalo).

Cache-Coherent Interconnects

For larger, discrete components, tight coupling is achieved via high-bandwidth, cache-coherent links such as CXL, UPI, or CAPI. These links allow an external accelerator to snoop on the CPU's cache and treat host memory as its own.

Roscoe et al. explored this paradigm by connecting a central processor to an FPGA via a cache-coherent link, removing the need for traditional peripheral drivers. This architectural shift allows accelerators to support low-latency transactional lookups alongside high-throughput analytical processing.

This interconnect strategy also transforms network infrastructure. Lazarev et al. integrated a network interface card (NIC) directly into the CPU's cache coherence domain via a memory interconnect. Instead of relying on high-latency PCIe doorbell transactions, the NIC detects new network requests simply by monitoring cache invalidation messages, drastically speeding up remote procedure calls (RPCs) in cloud environments.


The Power of Zero-Copy Execution

When hardware and software coordinate to eliminate data copies, application performance scales efficiently because processing units spend their time computing rather than moving bytes.

In real-time embedded systems, such as drone-borne radar imaging, Xu et al. utilized a unified memory approach to eliminate explicit data copying between the CPU and GPU, achieving critical latency reductions under strict power constraints. Similarly, for mobile computer vision, Kumar et al. leveraged a zero-copy shared memory architecture to eliminate the redundant copying of image frames and feature descriptors, enabling high-speed visual SLAM on low-power devices.

Traditional Pipeline:
[Load Image] -> [Copy to GPU] -> [GPU Kernel] -> [Copy to CPU] -> [Process Results]
                 ^^^^^^^^^^^^                     ^^^^^^^^^^^^
                 (Latency & Energy Bottlenecks)

Zero-Copy Pipeline:
[Load Image into Shared RAM] -> [GPU Kernel Processes In-Place] -> [CPU Reads In-Place]

This optimization is particularly vital for energy-constrained edge AI. Stylianopoulos et al. found that zero-copy memory mapping was the single most significant performance optimization for pattern matching on IoT devices, cutting energy consumption in half.

As applications grow to use multiple specialized accelerators simultaneously, unified memory acts as the orchestration layer. Hu implemented a shared memory management system using a ring buffer to coordinate multimodal AI models. Because the vision, audio, and text accelerators share a unified memory architecture, one accelerator can write its output embeddings directly into a memory slot accessible by the next. This completely bypasses CPU-mediated data copying, enabling long-duration inference on small, battery-powered devices.

Key Foundational Work

An asymmetric distributed shared memory model for heterogeneous parallel systems (2010) — contributed 3 concept elements to this direction.
Evaluation of streaming aggregation on parallel hardware architectures (2010) — contributed 2 concept elements to this direction.
Database Servers on Chip Multiprocessors: Limitations and Opportunities (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_virtual_memory_850cdb74,
  title = {unified virtual memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-virtual-memory/rdr_850cdb7475da422c8c7088e745974bae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_virtual_memory_850cdb74_v1,
  title = {unified virtual memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-virtual-memory/rdr_850cdb7475da422c8c7088e745974bae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake