GPU Residency for End-to-End Pipeline Acceleration

This direction is merged into another direction: data transfer latency, monolithic GPU execution. Its last version stays readable and citable here.

Version 1 of 127 papers37 conceptsfrozen 2026-10-04

In the early days of GPGPU (General-Purpose computing on Graphics Processing Units), the strategy was simple: identify the most computationally expensive loop in your code, port it to a CUDA or OpenCL kernel, and leave the rest on the CPU. This "offloading" model works well for isolated, massive matrix multiplications. However, as researchers pushed for real-time performance in complex systems—from radio telescopes to logic solvers—they hit a wall.

The bottleneck was no longer the math; it was the PCIe bus.

Modern research has shifted toward a more radical architecture: GPU residency. Instead of treating the GPU as a co-processor for specific tasks, these systems move the entire execution chain—including discrete logic, state management, and iterative loops—onto the graphics hardware. By keeping data resident in device memory from start to finish, these pipelines eliminate the latency of host-device synchronization.

The Cost of Conversation

To understand why residency matters, consider the "ping-pong" effect. In a traditional hybrid pipeline, the CPU prepares data, sends it to the GPU, waits for a result, and then decides what to do next. Even if the GPU computation takes microseconds, the overhead of moving data across the PCIe bus and synchronizing the two processors can take milliseconds.

Pedro Alonso and colleagues, working on border tracking in binary images, observed that the architectural benefit of data residency often outweighs the raw computational speed of the algorithms themselves. Similarly, Richard G. Edgar and Michael A. Clark demonstrated that for the Murchison Widefield Array, maintaining intermediate data within device memory was the only way to process 5 GiB/s of raw telescope data within a strict power budget.

Moving the "Boring" Parts to the GPU

The challenge of GPU residency is that GPUs are designed for high-throughput math, not complex branching logic or sequential updates. Achieving residency requires rethinking classic algorithms to fit the SIMT (Single-Instruction Multiple-Thread) model.

  1. Iterative Solvers and Logic: In Answer Set Programming (ASP), the solving loop involves complex conflict analysis. Agostino Dovier and Flavio Vella developed CUD@ASP, which keeps the entire solver state in device memory. By executing unit propagation and conflict analysis directly on the GPU, they avoided the frequent synchronization that usually kills performance in logic solvers.

  2. Adaptive Sampling: In numerical integration, algorithms like VEGAS+ update their sampling grids based on previous results. Anas Jnini and Roberto Passerone showed in cuVegas that performing these grid updates on the GPU—rather than offloading them to the CPU—eliminates the PCIe bottleneck, allowing the entire iterative loop to reside on the hardware.

  3. Optimization and Physics: Solving the Unit Commitment (UC) problem in power grids requires handling thousands of independent subproblems. Kibaek Kim and Youngdae Kim found that executing the entire optimization loop, including nonlinear physics and discrete decision logic, directly on the GPU shifted the bottleneck from algorithmic complexity to hardware memory bandwidth, enabling real-time tracking of massive power systems.

Closing the Loop in Computer Vision

Computer vision has been a primary beneficiary of this trend. While neural networks are naturally GPU-friendly, the "pre" and "post" processing steps often remain on the CPU, creating a drag on the system.

Ning Kang and Shutao Xia addressed this in PILC, a lossless image compression framework. By integrating the entropy coder directly onto the GPU, they ensured that the high-speed predictions of their neural model weren't negated by sequential CPU post-processing. Similarly, Liang Cao and Peng Jia built a pre-processing framework for astronomy that loads raw data once and performs alignment, subtraction, and detection entirely within GPU memory.

This "load once, process many" philosophy is also seen in:

  • 3D Reconstruction: Konstantinos Tzevanidis and Xenophon Zabulis offloaded every stage from segmentation to texture mapping, allowing 30 FPS "4D video" capture.
  • Medical Imaging: Lukas Fisch and Anna Kraus enabled ultra-large-scale neuroimaging analysis by moving the entire Voxel-based Morphometry pipeline to the GPU.
  • Feature Matching: Christopher Parker and Gil Levi achieved "breakneck" speeds in CLATCH by keeping detection, extraction, and matching all within graphics memory.

The Future of Numerical Pipelines

Even fundamental linear algebra is being rewritten for residency. Evelyne Ringoot, Alan Edelman, and Valentin Churavy have focused on Singular Value Decomposition (SVD). They argue that numerical pipelines relying on the CPU for final steps—like the transition from bidiagonal form to singular values—introduce fatal synchronization overhead. Their work on fully GPU-resident bidiagonalization demonstrates that even memory-bound "middle steps" can achieve 100x speedups when they never have to leave the GPU context.

The intuition is clear: in the era of massively parallel hardware, the fastest way to complete a task is often to do it entirely on the "slower" processor (in terms of logic) to avoid the cost of moving the data back to the "faster" one. By treating the GPU as a self-contained universe for data, researchers are finally unlocking the throughput necessary for the next generation of real-time scientific and industrial applications.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gpu_residency_for_end_to_end_pipeline_ac_f7b3eb13,
  title = {GPU Residency for End-to-End Pipeline Acceleration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gpu-residency-for-end-to-end-pipeline-acceleration/rdr_f7b3eb13beae4be292a7ddcacea0ed18},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_gpu_residency_for_end_to_end_pipeline_ac_f7b3eb13_v1,
  title = {GPU Residency for End-to-End Pipeline Acceleration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gpu-residency-for-end-to-end-pipeline-acceleration/rdr_f7b3eb13beae4be292a7ddcacea0ed18/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}