UniPipe: Unified Inference Serving and Fine-Tuning for Deadline-Constrained DNN Surrogate Workflows

Jaiaid Mobin, Avinash Maurya, M. Mustafa Rafique, Bogdan NicolaearXiv 2026 · arXiv

TL;DR

UniPipe is a novel system that enables deep learning models to "learn while doing" on a single edge GPU without missing critical real-time deadlines. By fusing inference and fine-tuning into a single execution context and using a smart scheduler, it achieves up to 50% better accuracy and significantly more stable performance than traditional methods for high-speed scientific imaging.

What Problem Does This Paper Address?

In modern scientific experiments, such as X-ray ptychography (a high-resolution imaging technique), sensors produce data at staggering rates—up to 300 Gbps. To make sense of this data in real-time and steer the experiment, researchers use Deep Neural Network (DNN) "surrogates" that are 100–1000x faster than classical math-heavy solvers.

However, there is a catch: as an experiment progresses, the nature of the samples changes, causing the pre-trained DNN to lose accuracy (model drift). To fix this, the model needs continual fine-tuning on fresh data.

The standard approach is to run two separate processes on the GPU: one for inference (serving) and one for training (fine-tuning). This creates "multi-context contention," where the two processes fight for the GPU's resources. This conflict causes unpredictable latency spikes. In a workflow with hard deadlines—where a result is useless if it arrives late—these spikes lead to frequent "deadline misses," forcing the system to fall back on low-quality estimates.

Fig. 1: DNN surrogate workflow for Ptychographic reconstruc- tion at the edge. A detector produces diffraction patterns at rate drate; an edge node must reconstruct each pattern within a per-pattern deadline D using either a fast DNN surrogate (lower accuracy) or a slow classical solver (higher accuracy).

Methods

The authors introduce UniPipe, which eliminates GPU contention by unifying inference and training into a single CUDA context. Its design is built on three main pillars:

  1. Fused Forward Pass with Selective Backward Pass: Instead of running separate passes, UniPipe combines unlabeled inference requests and labeled training samples into a single batch. It runs the forward pass once for everyone, then performs the expensive backward pass (backpropagation) only on the training samples.
  2. Exploiting Non-linear Latency: Modern GPUs have a "saturation knee." Below a certain batch size, adding more samples is essentially "free" because the GPU has idle processing units. UniPipe profiles this curve to maximize work without increasing time.
  3. DP-Based Offline Scheduling: Since scientific sensor streams are often repetitive and predictable, UniPipe uses a polynomial-time dynamic-programming (DP) scheduler. This scheduler pre-calculates the optimal batch composition (how many training vs. inference samples) to ensure every inference request meets its deadline while maximizing the amount of fine-tuning performed.
Fig. 5: Forward- and backward-pass latency of PtychoNN as a function of batch size. It increases visibly only after saturating the GPU parallelism, which allows batching up to a certain threshold for free (Lower is better).

Datasets

The system was evaluated using two primary ptychographic datasets:

  • Dataset 1: A benchmark set of ~26,000 diffraction patterns from the PtychoNN repository, covering an 1800nm x 1800nm object.
  • Dataset 2: A larger set of ~35,000 scans collected at the Advanced Photon Source (APS), featuring a random etched pattern in tungsten.

Key Insights and Ideas

  • The Contention Tax: The authors quantified that simply running training and inference side-by-side (multi-context) inflates latency by ~2x and creates massive variability. This variability is the "silent killer" of real-time systems because schedulers must plan for the worst-case scenario.
  • Single-Context Stability: By forcing everything into one sequence (single-context), UniPipe makes latency predictable again. As shown in the paper's evaluation, the latency spread was reduced from 106% of the mean down to just 57%.
  • Fused Efficiency: Fusing the forward pass isn't just about avoiding contention; it's about efficiency. It eliminates the redundant computation that would occur if training and inference were handled separately.
Fig. 2: Effect of online fine- tuning on inference accuracy. Lower is better.

Key Results

UniPipe was tested against a "Pretrained" baseline (no learning) and a "Multi-Context" baseline (standard concurrent learning).

  • Accuracy: UniPipe reduced inference Mean Squared Error (MSE) by up to 50% and overall reconstruction MSE by up to 30% compared to the pretrained model.
  • Reliability: It reduced the deadline-miss rate by 13%–22% compared to the multi-context approach.
  • Throughput: UniPipe demonstrated higher total throughput (training + inference) by better utilizing the GPU's parallel architecture without the overhead of context switching.

Limitations and Future Work

While UniPipe is a significant step forward for edge computing in science, the authors note several limitations:

  • Single-GPU Focus: The current design is optimized for a single edge device and does not yet address scaling across multiple GPUs.
  • Model Architecture Constraints: It assumes the model's forward pass is "mode-invariant." Models that behave differently during training vs. inference (like those using BatchNorm or Dropout) would require additional profiling or separate sub-batches.
  • Static Scheduling: The scheduler assumes that the latency curves and the classical solver's throughput remain stable. If these drift significantly, the scheduler would need to be re-run.

Future work includes extending UniPipe to more complex architectures like Transformers and developing adaptive methods for selecting which specific data points are most valuable for fine-tuning.

Summary updated

Abstract

DNN surrogates are increasingly used to accelerate scientific workflows that must process high-rate sensor streams in near real time, exemplified by Ptychographic imaging at next-generation X-ray light sources. Such surrogates deliver 100-1000x higher throughput than iterative solvers, enabling faster steering of experiments. To preserve surrogate accuracy during an experiment, as the input distribution evolves, state-of-the-art deployments freeze the pretrained surrogate or run a separate CUDA context that fine-tunes concurrently with serving. However, multi-context concurrency causes significant contention on the GPU, inflating both the mean and the variance of inference latency, triggering deadline misses in workflows where sensor data has to be processed within a hard per-unit deadline. To address this challenge, we propose UniPipe, a single-context system that unifies inference serving and fine-tuning of a DNN surrogate on a single edge GPU. UniPipe is driven by three key insights: both serving and fine-tuning require a forward pass and can be fused into one; the per-sample latency of modern GPUs is non-linear in batch size; and the workload is repetitive across intervals, enabling offline scheduling. Specifically, UniPipe fuses training and inference data in a single forward pass with a selective backward pass on the training subset, and uses a polynomial-time scheduler that exploits the latency curve to compute the batch composition. We evaluate UniPipe on a real-world Ptychographic reconstruction workflow with sensor rates up to 5000 patterns/sec.; UniPipe lowers reconstruction MSE by up to 30% and inference MSE by up to 50% over a pretrained baseline, and reduces the deadline-miss rate by 13%-22% over a multi-context baseline. UniPipe's unified deployment reduces the latency spread from 106% to 57% of the mean, and increases total throughput compared to multi-context baseline.
BibTeX
@misc{mobin2026unipipe,
  author = {Jaiaid Mobin and Avinash Maurya and M. Mustafa Rafique and Bogdan Nicolae},
  title = {{UniPipe: Unified Inference Serving and Fine-Tuning for Deadline-Constrained DNN Surrogate Workflows}},
  year = {2026},
  eprint = {2611.14748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14748},
}
Figure