Hybrid Co-Execution across Heterogeneous Compute Units

113 concepts101 papers2 proposals

In the pursuit of high-performance computing, the modern systems paradigm has shifted heavily toward acceleration. Whether training large language models, running genomic sequence alignments, or processing real-time video streams, standard practice dictates offloading the heavy lifting to a specialized accelerator—typically a GPU, NPU, or FPGA.

However, this accelerator-centric design pattern introduces a severe architectural irony: by treating the host CPU merely as a traffic cop or data loader, systems leave a massive portion of their total computational capacity completely idle. This binary execution model creates a pervasive efficiency gap across modern hardware stacks, from mobile systems-on-chips (SoCs) to exascale datacenters.

To overcome this, a growing body of research focuses on hybrid co-execution—rearchitecting runtimes and schedulers to treat all available hardware units as peer compute nodes, dynamically partitioning workloads, and slicing tasks to eliminate resource underutilization.


The Anatomy of Underutilization

Resource underutilization manifests in two primary ways: host-accelerator idling and internal hardware slack.

Host-Accelerator Idling

In traditional offloading frameworks, execution is strictly sequential or highly asymmetric. As M. Hernández et al. observe in diffusion MRI pipelines, the CPU frequently sits idle, serving merely as a monitor while the GPU performs the bulk of the computation. Guo-Qing Lei et al. document the same pattern in bioinformatics applications, noting that exclusive device allocation wastes a substantial portion of a heterogeneous system's available capacity.

This idling cuts both ways. In workflows like active learning for physical simulations, the pipeline often reverts to serial execution: high-performance GPUs sit entirely inactive while CPU-intensive data labeling or oracle calculations are performed, as detailed in PAL – parallel active learning for machine-learned potentials.

Internal Slack and Fragmentation

Even when an accelerator is actively processing a workload, standard hardware metrics can be deeply misleading. As shown in Towards Pareto Optimal Throughput in Small Language Model Serving, high reported GPU activity often masks low streaming multiprocessor (SM) occupancy. A device may appear 100% busy, yet its internal processing threads are stalling because the workload fails to fill the hardware's scheduling gaps or is bottlenecked by memory bandwidth.

Furthermore, coarse-grained resource management exacerbates this waste. In multi-tenant cloud environments, assigning entire accelerators or fixed hardware partitions (such as NVIDIA's Multi-Instance GPU) to single tasks creates "internal slack." Workloads rarely match these rigid, discrete resource sizes perfectly, leaving non-contiguous portions of the compute engine fragmented and unusable by concurrent tasks (Myeongjae Jeon et al.; Munkyu Lee).


Elevating the Host to a Peer Compute Node

The core insight of hybrid co-execution is that the host processor should not be relegated to management tasks. Instead, systems can maximize throughput by distributing workloads across both the CPU and the accelerator simultaneously.

Traditional Offloading:
[ CPU (Idle/Control) ] -------> [ GPU (100% Saturated) ]

Hybrid Co-Execution:
[ CPU (Compute Node) ] ---\
                           ===> [ Shared Workload Pool ]
[ GPU (Compute Node) ] ---/

To achieve this without introducing massive synchronization overhead, frameworks must intelligently partition execution graphs based on what each hardware unit does best. For deep neural network (DNN) inference, Jean-François David et al. introduce StarONNX, a system that uses graph-partitioning tools to split models at runtime, routing segments to both multi-core CPUs and GPUs to prevent hardware saturation. A similar approach is championed by the authors of Exploiting Processor Heterogeneity to Improve Throughput and Reduce Latency for Deep Neural Network Inference, breaking the "GPU-only" habit by turning underutilized CPUs into active inference nodes.

This synergy is equally effective when pairing CPUs with other specialized hardware:

  • CPUs and FPGAs: Alessandro Capotondi et al. demonstrate that treating an ARM CPU and an FPGA fabric as synergistic partners allows the CPU's vector engines to handle memory-bound neural network layers while the FPGA focuses on compute-bound layers, closing the efficiency gap.
  • GPUs and NPUs: On modern mobile SoCs, applications typically utilize only a single component at a time (Anuj Pathania et al.). To break this barrier during large-scale model inference, Nikhil Bhendawade et al. parallelize speculative decoding by running the draft and target models concurrently across the NPU and GPU, using early-exit signals to keep both accelerators in lockstep.

Fine-Grained Slicing and Workload Co-Location

When a single workload cannot be split across different physical devices, the alternative is to pack multiple workloads onto the same device. However, naive co-location causes severe interference and latency spikes. Co-execution frameworks solve this by slicing tasks temporally or spatially at a microscopic level.

An individual GPU kernel is typically bottlenecked by either computation or memory access, leaving the alternative resource underutilized. Jianlong Zhong addresses this by dynamically "slicing" large kernels into smaller sub-units and using a performance model to co-schedule complementary slices—pairing a compute-heavy task with a memory-heavy one to fill idle hardware cycles.

Unscheduled Execution:
Kernel A (Compute-Bound): [=== Compute ===][... Idle Mem ...]
Kernel B (Memory-Bound):  [... Idle Comp ...][=== Memory ===]

Slices Co-Scheduled via Kernelet:
Resource Utilization:     [=== Compute A ===][=== Memory B ===]
                          [... Memory B ...][... Compute A ...]

For multi-tenant scenarios where latency-critical tasks (like interactive chatbots) must share space with throughput-oriented background tasks (like batch offline processing), rigid segregation is highly inefficient. Wei Zhao et al. introduce Tally, which breaks monolithic GPU kernels into fine-grained thread blocks. By scheduling at the thread-block level rather than the kernel level, the system achieves robust performance isolation, allowing background training workloads to opportunistically harvest the idle capacity of high-priority inference tasks without triggering latency violations.


Asynchronous Pipelining and Resource Borrowing

A final, powerful facet of hybrid co-execution involves using the idle capacity of one resource to completely hide the latency bottlenecks of another.

In distributed Graph Neural Network (GNN) training, the primary bottleneck is network communication—waiting to fetch node features from remote servers. Aishwarya Sarkar et al. exploit underutilized host CPU cores to run an asynchronous prefetching and eviction scheme. While the GPU is busy computing the current layer, the idle CPU "looks ahead" in the graph, prefetching and caching remote features so the GPU never stalls waiting for I/O.

This concept of "resource borrowing" can even extend to the hardware interconnects themselves. Yichao Yuan introduces Vortex, a data analytics engine that overcomes memory capacity limits by aggregating the I/O bandwidth of every GPU in a multi-GPU system. When a specific GPU is deep in a compute-heavy AI task, its host-to-device PCIe link is largely idle. Vortex "borrows" these idle PCIe lanes from inactive or compute-bound GPUs to stream data to the active analytics engine, turning wasted interconnect capacity into an aggregated high-speed data pipeline.

By shifting from an exclusive, accelerator-centric mindset to a holistic, hybrid co-execution model, modern software frameworks can reclaim the hidden capacity trapped within heterogeneous hardware, delivering massive throughput gains without requiring a single watt of additional physical compute infrastructure.

Key Foundational Work

CPU-GPU hybrid accelerating the Zuker algorithm for RNA secondary structure prediction applications (2012) — contributed 1 concept elements to this direction.
Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_resource_underutilization_bc325c44,
  title = {heterogeneous resource underutilization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-resource-underutilization/rdr_bc325c44336748e7984077fd9b371686},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_resource_underutilization_bc325c44_v1,
  title = {heterogeneous resource underutilization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-resource-underutilization/rdr_bc325c44336748e7984077fd9b371686/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake