Abstract
Production deep learning clusters often face a resource imbalance: high-performance training GPUs (e.g., NVIDIA V100s) are frequently oversubscribed, while inference-oriented GPUs (e.g., T4s) remain idle during off-peak hours. Utilizing these heterogeneous resources for distributed training—known as hybrid-device training—presents significant challenges due to disparities in compute capability and memory capacity. We propose QSync, a training system that enables efficient synchronous data-parallel training over hybrid devices by strategically exploiting quantized operators. QSync selects a quantization-minimized configuration for each device, minimizing accuracy degradation while maintaining the efficiency gains of low-precision math. Our system features a bi-directional mixed-precision predictor, a neighborhood-aware cost replayer, and an allocator that synchronizes workers with minimal quality loss. Experiments demonstrate that QSync achieves a 0.27–1.03% accuracy improvement over uniform precision methods while maintaining high throughput.
Introduction: The Hybrid Device Challenge
In modern AI clouds, hardware is typically bifurcated into training clusters (A100s, V100s) and inference clusters (T4s, A10s). While inference GPUs are plentiful, they are often underutilized outside of peak serving hours. Integrating these "weaker" GPUs into training pipelines is an attractive way to increase throughput, but it introduces the "straggler" problem. In standard synchronous data-parallel training, the entire cluster must wait for the slowest device to complete its gradient computation.
Existing solutions often fall short. Dynamic batch sizing (assigning smaller batches to weaker GPUs) alters training semantics and can degrade final model accuracy. Uniform low-precision quantization (forcing all devices to INT8) speeds up the cluster but sacrifices significant model quality. QSync addresses this by using precision as a load-balancing mechanism, finding the "Goldilocks" zone of quantization: the minimum amount of low-precision math required to keep the cluster synchronized without unnecessary accuracy loss.
The QSync Framework
QSync introduces a quantization-minimized approach through three core components designed to bridge the gap between PyTorch's computational graph and an optimized backend.
1. The Predictor and Replayer
To make informed decisions about which layers to quantize, QSync must first understand the costs and benefits. The Indicator uses a bi-directional mixed-precision theory to estimate the sensitivity of specific DNN layers to quantization in both forward and backward passes.
Crucially, QSync accounts for the "Casting Tax." Converting a tensor from FP32 to INT8 is not a zero-cost operation. The Replayer utilizes a neighborhood-aware cost mapper to simulate the execution timeline, explicitly modeling these conversion overheads and communication dependencies. This allows QSync to avoid "false optimizations" where the time saved by quantization is eclipsed by the time spent on data conversion.
2. The Allocator
The Allocator solves the optimization problem of precision assignment. It begins by placing inference GPUs in their fastest possible low-precision mode to satisfy memory and speed constraints. It then performs a "strategic recovery," greedily upgrading the precision of the least sensitive operators one by one. This process continues as long as the GPU remains within its memory limits and does not exceed the synchronization window set by the faster training GPUs. Effectively, QSync fills the "waiting window" of the faster devices with higher-quality computation on the slower devices.
3. LP-PyTorch Backend
To support these operations, the authors developed a customized backend for PyTorch. This backend supports a wide variety of low-precision kernels (INT8, FP16) across different GPU architectures and includes optimizations like Dequantization Fusion to minimize the overhead of moving between numerical formats.
Experimental Evaluation
QSync was evaluated using industry-standard benchmarks, including ImageNet (VGG16, ResNet50), SQuAD (BERT), and SWAG (RoBERTa).
Accuracy and Throughput
QSync achieved a consistent 0.27% to 1.03% accuracy improvement over uniform low-precision schemes. Unlike dynamic batch sizing, which struggles with semantic shifts caused by varying local batch sizes, QSync maintains consistent training math across the cluster. Furthermore, the system maintained high training efficiency, matching the throughput of uniform precision while delivering superior model quality.
Prediction Fidelity
The QSync Predictor proved highly effective, simulating distributed training timelines with less than 5% error. This significantly outperforms previous state-of-the-art predictors (such as DPro) that ignore the critical costs of precision casting. Additionally, the LP-PyTorch backend provided a >10% end-to-end performance gain for INT8 training compared to standard implementations.
Conclusion and Future Work
QSync demonstrates that precision is a powerful lever for load balancing in heterogeneous clusters. By treating quantization as a resource to be minimized rather than a uniform constraint, QSync allows researchers to harness idle inference hardware without the typical accuracy penalties.
Future work will focus on reducing the initial profiling overhead (currently requiring ~50 iterations) and extending the quantization-minimized logic beyond data-parallel training to tensor-parallel and pipeline-parallel configurations. The researchers also intend to investigate whether simple "flooring" can replace complex stochastic quantization to further streamline training quality restoration.
Summary updated