DataHelix: A CPU-GPU Co-Scheduling Data Plane for Multimodal Model Training

Yuanhe Li, Jiazhen Wu, Dian Ding, Guangtao Xue, Yiming ZhangarXiv 2026 · arXiv

TL;DR

DataHelix is a specialized CPU-GPU co-scheduling data plane designed to eliminate the massive "pipeline bubbles" that plague multimodal model training. By treating GPU compute windows as deadlines for data readiness and making routing decisions based on lightweight metadata, it achieves up to a 2.72× throughput boost and an 87% reduction in tail latency compared to traditional data loaders.

What Problem Does This Paper Address?

In the world of deep learning, we’ve long relied on a "loose coupling" between data loading and model execution. For homogeneous text or image tasks, simply giving every GPU the same number of samples usually keeps the cluster balanced.

Multimodal training (e.g., long-video understanding or Diffusion Transformers) shatters this assumption. A single batch might contain one 2-second clip and one 60-second high-res video. This creates two types of heterogeneity:

  1. CPU Skew: Decoding and preprocessing a long video takes orders of magnitude more time than a short one.
  2. GPU Skew: Different samples result in vastly different numbers of visual tokens, meaning some GPUs finish their compute steps much faster than others.

In synchronous training, these issues compound. If even one "straggler" rank is delayed by slow preprocessing, every other GPU in the cluster sits idle at the synchronization barrier. As shown in Figure 2, a single rank's data wait can be hundreds of times longer than its neighbors, dragging down global efficiency.

Figure 2: Qwen-VL rank skew and long waits in native synchronous training.

Methods

DataHelix moves away from optimizing for aggregate CPU throughput and instead focuses on per-rank data readiness. Its architecture is built on four key pillars:

  • Metadata-First Control: Decisions are made at the "descriptor" stage—when a sample is just a file path and some basic metadata—before it is decoded into a massive tensor. This avoids the "data amplification" problem where a few kilobytes of metadata turn into gigabytes of RGB pixels.
  • Deadline-Aware Scheduling: The system treats the time a GPU takes to process batch $t$ as the "readiness budget" for batch $t+1$. A central Coordinator predicts the completion time for each sample across available DataServers (local and remote) and routes tasks to minimize deadline misses (Figure 5).
  • Two-Tier Load Balancing:
    1. Tier 1 (Pre-balancing): Uses token-count estimates to build batches that equalize the expected GPU compute load.
  • Tier 2 (DDLB): A "cost-gated" rebalancing mechanism that corrects residual skew after data is materialized. Crucially, it only moves data if the predicted gain exceeds the transfer cost, and it uses high-speed collectives (HCCL/NCCL) to move tensors at accelerator-interconnect speeds.
  • Asynchronous Prefetch: A deep pipeline that overlaps descriptor dispatch, remote execution, and tensor delivery with the current GPU step (Figure 6).
Figure 6: Asynchronous input preparation and ordered delivery overlap GPU execution.

Datasets

The authors evaluated DataHelix across 11 workloads using:

  • Kinetics: Used for video training with various decode profiles (small, mid, large, and mixed).
  • OpenVid: A dataset for high-resolution video-text tasks.
  • Models: Evaluation included VideoMAE, Wan-1.3B, and Qwen-VL-3B (for multimodal SFT/PT).

Key Insights and Ideas

  • The "Data Amplification" Asymmetry: This is the paper's "aha moment." In video pipelines, data grows by orders of magnitude during preprocessing. Therefore, placement decisions must be finalized at the descriptor stage. Deferring load balancing until after decoding is often too expensive due to network costs.
  • GPU Progress as a Clock: Instead of treating the data pipeline as an independent producer, DataHelix treats the GPU as the master clock. The goal isn't to be as fast as possible; it's to be "fast enough" to meet the next GPU compute window.
  • Bypassing the CPU Network Stack: By using NCCL/HCCL for post-materialization rebalancing, DataHelix can redistribute gigabyte-scale tensors in under a second—a 99% reduction in latency compared to standard gRPC/TCP methods.

Key Results

DataHelix was tested on an 8-card Ascend 910B server and 4×2 A100 GPU clusters. Key findings include:

  • Throughput: Achieved a 2.72× geometric-mean throughput improvement over standard PyTorch DataLoaders and 1.82× over a Ray actor pool under the same CPU budget.
  • Latency: Reduced mean p95 input-wait latency by 87.0%.
  • Deadline Compliance: Reduced the mean deadline-miss rate by 67.8%.
  • Scaling: Unlike native loaders, which see performance degrade as batch sizes increase, DataHelix maintains stable throughput and limits tail waiting even at larger batch sizes (Figure 13).

Limitations and Future Work

  • Profiling Overhead: The system relies on offline profiles to estimate CPU decoding costs and GPU token counts. While it uses online heartbeats to refine these, highly dynamic or unseen data distributions might require more robust online estimation.
  • Synchronous Focus: The current design is optimized for synchronous data-parallel training. While this is the industry standard for large models, exploring how these co-scheduling principles apply to asynchronous or pipeline-parallel setups is a logical next step.
  • Framework Integration: While already integrated into PyTorch and MindSpore, the "metadata-first" requirement may require users to slightly adapt how they define their Datasets to expose cost estimates.

Summary updated

Abstract

Multimodal model training breaks the loose coupling between data loading and model execution assumed by conventional training pipelines. Large per-sample variation in decoding cost, token/latent count, and tensor size causes both CPU preprocessing time and GPU compute time to vary substantially across data-parallel ranks (training processes). In synchronous training, these two sources of heterogeneity interact tightly: GPU execution determines how long the input pipeline has to prepare the next batch, while delayed preprocessing in any rank in turn stalls all GPUs at synchronization. As a result, optimizing CPU throughput or GPU load balance in isolation cannot eliminate pipeline bubbles. We present DataHelix, a metadata-first CPU-GPU co-scheduling data plane for multimodal training. DataHelix jointly reasons about preprocessing demand and GPU execution progress, treating each GPU compute window as the readiness budget for the next batch and coordinating preprocessing across ranks to minimize deadline misses rather than CPU makespan. To avoid expensive data movement, placement is decided while samples remain lightweight descriptors, before decoding and collation amplify them into large tensors. Descriptor-level token-latent-aware batching performs coarse pre-balancing, while cost-gated collective rebalancing corrects residual skew only when its expected benefit exceeds transfer cost. Across 11 video and multimodal workloads on 8 Ascend 910B accelerators, DataHelix reduces mean deadline-miss rate by 67.8% and mean p95 input-wait latency by 87.0%, while achieving 2.72x geometric-mean throughput over rank-local DataLoaders and 1.82x over a Ray actor pool under the same CPU budget.
BibTeX
@misc{li2026datahelix,
  author = {Yuanhe Li and Jiazhen Wu and Dian Ding and Guangtao Xue and Yiming Zhang},
  title = {{DataHelix: A CPU-GPU Co-Scheduling Data Plane for Multimodal Model Training}},
  year = {2026},
  eprint = {2611.09411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09411},
}
Figure