For years, the standard blueprint for high-performance computing has treated the GPU as an accelerator or a "coprocessor." In this traditional model, the CPU acts as the brain: it manages the application's control logic, orchestrates data structures, prepares a batch of data, and shuffles it over the PCIe bus to the GPU. The GPU then performs massive parallel math and sends the results back across the bus for the CPU to handle the next step.
However, as GPUs have grown exponentially faster, this hybrid model has run into a hard physical wall: the PCIe bottleneck, often called the "PCIe tax." When an algorithm requires frequent interactions between the host CPU and the device GPU, the time spent moving data back and forth completely obliterates the computational speedup.
To break through this barrier, a powerful paradigm shift has emerged: end-to-end GPU-resident processing. Instead of using the GPU as a temporary math sandbox, researchers are moving the entire computational pipeline—including control loops, data manipulation, state management, and post-processing—directly onto the device.
The Core Intuition: Suboptimal GPU Execution Beats PCIe Latency
The fundamental insight behind end-to-end GPU execution is a counterintuitive trade-off: It is often faster to execute sequential or poorly parallelized code on a GPU thread than it is to transfer data back to a fast CPU core.
Consider a data compression pipeline. In a traditional hybrid framework, a GPU might handle parallel pattern matching, but a CPU handles the final sequential encoding. As shown by Zhang et al., offloading the encoding step back to the CPU creates a massive communication bottleneck. By moving the entire compression pipeline—including the final encoding—directly onto the GPU, their framework achieved up to a 272x speedup. Similarly, in neural image compression, Kang et al. demonstrated that integrating the entropy coder directly on the GPU prevents high-speed neural model predictions from being negated by sequential CPU post-processing.
When data remains resident in high-bandwidth GPU memory, the application shifts from being communication-bound to being memory-bandwidth-bound. As Shanbhag et al. demonstrated in database analytics, treating the GPU as the primary execution engine where the entire working set resides drastically outperforms the coprocessor model, allowing query execution to exceed traditional CPU-GPU performance limits.
Keeping the Control Loop On-Device
Many real-world problems are iterative, requiring a control loop to check for convergence, analyze conflicts, or update states. If the CPU manages this loop, it must poll the GPU for results at every single iteration.
Traditional Hybrid Model:
[CPU: Control Logic] --(PCIe Transfer)--> [GPU: Math] --(PCIe Transfer)--> [CPU: Check Convergence] -> Loop
GPU-Resident Monolithic Model:
[CPU: Launch Kernel]
|
v
[GPU: Allocate Memory -> Run Math -> Check Convergence -> Update State] -> Loop Entirely On-Device
To eliminate this overhead, researchers are rewriting complex solver loops to run monolithically on the GPU:
- Logic Solvers: Dovier et al. developed an Answer Set Programming (ASP) solver that runs its entire unit propagation and conflict analysis loop directly on the GPU, maintaining the solver state entirely in device memory.
- Mathematical Optimization: For massive batches of non-linear programming subproblems, Kim et al. kept both the control logic and numerical computations within the GPU environment. When applied to power grid optimization, Kim et al. showed that keeping the iterative loop on-device ensures that execution speed is limited only by internal memory bandwidth rather than communication latency.
Fusing Preprocessing and Post-Processing
Even if the core algorithm runs entirely on the GPU, performance can still leak away if the input data must be staged or manipulated on the CPU first.
For instance, training time-series models requires creating sliding windows of data. Koad et al. addressed this by performing data manipulation directly on the GPU using native operations and zero-copy views, bypassing intermediate CPU processing entirely.
The same principle applies to scientific computing and basic data reductions. In exascale numerical libraries, Brown demonstrated that performing matrix assembly directly on the GPU, rather than staging it through the CPU, is vital for maximizing hardware utilization and enabling asynchronous execution. In spatial data processing, Deussen et al. offloaded both the distance calculations and the final minimum-value reduction pipeline entirely to the graphics hardware, achieving significant speedups by eliminating post-processing on the CPU.
Native State and Hypothesis Management
To make a pipeline truly GPU-native, the device must be capable of managing its own internal state and complex data structures dynamically, without calling home to the CPU. This requires rethinking how memory is allocated and how hypotheses are tracked in parallel:
- Dynamic Data Structures: Hundt et al. designed high-performance hash tables that execute operations entirely within GPU kernels, unlocking fast, host-independent data pipelines.
- Hypothesis Tracking: In speech recognition decoding, Chen et al. performed lattice generation and pruning entirely within GPU memory using pre-allocated vectors to store alternative hypotheses, avoiding frequent host-device synchronization.
- Pipeline Autonomy: In randomized Singular Value Decomposition (SVD), Struski et al. built an end-to-end pipeline that handles everything from on-device random number generation to final decomposition, completely avoiding the PCIe tax. Similarly, Parker et al. maintained an entire computer vision feature processing pipeline (detection, extraction, and matching) within GPU memory to achieve sub-microsecond processing times.
This autonomy is especially critical in distributed or real-time streaming environments. In GPU clusters, Huang et al. minimized communication overhead by pre-allocating GPU buffers and executing reduction operations directly on the device. At the extreme scale of high-energy physics, Aaij et al. demonstrated that a complete high-level trigger system can process 40 Tbit/s of raw detector data by containing the entire event-processing sequence within a network of GPUs.
Conclusion
The transition to end-to-end GPU-resident processing represents a maturity milestone in accelerator computing. By treating the GPU not as a fast calculator but as a self-contained execution environment, these architectures trade away the raw sequential speed of the CPU to eliminate the far more damaging latency of the PCIe bus. The result is a generation of software that fully unleashes the massive throughput of modern parallel hardware.
Key Foundational Work
A Graphics Hardware Accelerated Algorithm for Nearest Neighbor Search (2006) — contributed 1 concept elements to this direction.
Parallel Execution of the ASP Computation - an Investigation on GPUs (2015) — contributed 1 concept elements to this direction.