QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices

Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Yibo Zhu, Chuan Wu

TL;DR

QSync is a novel distributed training system designed to utilize idle inference GPUs (like NVIDIA T4s) alongside powerful training GPUs (like V100s) in a single synchronous cluster. By strategically applying quantization (low-precision math) only where necessary to match the speed and memory constraints of the slower devices, QSync eliminates the "straggler" problem and preserves model accuracy better than standard uniform-precision or dynamic batch-sizing methods.

What Problem Does This Paper Address?

Production AI clouds are often split into two worlds: high-performance training clusters (A100s, V100s) that are constantly backlogged, and inference clusters (T4s, A10s) that sit idle during off-peak hours.

"Hybrid-device training" aims to use these idle inference GPUs to speed up training jobs. However, inference GPUs typically have significantly less memory and lower compute throughput than training GPUs. In standard synchronous data-parallel training, the entire cluster moves at the speed of the slowest device. If you use the same settings across all devices, the inference GPUs become massive bottlenecks (stragglers).

Previous attempts to fix this used dynamic batch sizing (giving smaller batches to weaker GPUs), but this changes the training semantics and often degrades final model accuracy. Others used uniform low-precision (e.g., forcing everything to INT8), which speeds up the weak GPUs but sacrifices too much model quality. QSync seeks a "Goldilocks" zone: the minimum amount of quantization needed to keep the cluster synchronized without wasting accuracy.

Methods

QSync introduces a "quantization-minimized" approach through three main components:

  1. The Predictor: This consists of two parts. First, an Indicator uses a bi-directional mixed-precision theory to estimate how sensitive specific DNN layers are to quantization in both forward and backward passes. Second, a Replayer uses a "neighborhood-aware cost mapper" to simulate the execution timeline, accounting for the overhead of "casting" (converting data between precisions) and communication dependencies.
  2. The Allocator: This module solves an optimization problem. It starts by putting the inference GPUs in the fastest possible low-precision mode to fit the required batch size. It then greedily "recovers" (upgrades) the precision of the least sensitive operators one by one, as long as the GPU stays within its memory limits and doesn't slow down the overall cluster throughput.
  3. LP-PyTorch Backend: The authors built a customized backend for PyTorch that supports a wide variety of low-precision kernels (like INT8 and FP16) across different GPU architectures. It includes optimizations like "Dequantization Fusion" to reduce the overhead of moving between different numerical formats.

Datasets

The researchers evaluated QSync using several industry-standard benchmarks:

  • ImageNet: Used for training VGG16 and ResNet50 from scratch.
  • SQuAD: Used for fine-tuning BERT for question-answering tasks.
  • SWAG: Used for fine-tuning RoBERTa for multiple-choice tasks.

Key Insights and Ideas

  • Precision as a Load Balancer: Instead of changing the number of samples a GPU processes (batch size), QSync changes the complexity of the math it performs. This keeps the training math (gradients, batch norm statistics) consistent across the cluster.
  • The "Casting" Tax: Converting a tensor from FP32 to INT8 isn't free. QSync’s Replayer is unique because it explicitly models these conversion costs. Sometimes, quantizing a single layer makes the overall training slower because the time spent converting data outweighs the time saved in computation.
  • Strategic Recovery: The core "aha" moment is that because synchronous training is limited by the slowest device, there is often a "waiting window." QSync uses this window to upgrade as many layers as possible back to high precision on the inference GPUs, effectively "filling" the idle time with higher-quality computation.
  • Layer Sensitivity: Not all layers are created equal. The paper finds that layers deeper in the network often show significantly higher sensitivity to quantization than early layers.

Key Results

  • Accuracy: QSync achieved a 0.27% to 1.03% accuracy improvement over uniform low-precision schemes. It also outperformed dynamic batch sizing, which struggled with the semantic shifts caused by varying local batch sizes.
  • Throughput: The system maintained high training efficiency, matching the speed of uniform precision while delivering better model quality.
  • Prediction Accuracy: The QSync Predictor was highly effective, simulating distributed training timelines with less than 5% error, whereas previous state-of-the-art predictors (like DPro) had much higher error rates because they ignored casting costs.
  • Kernel Efficiency: Their LP-PyTorch backend provided a >10% end-to-end performance gain for INT8 training compared to standard implementations.

Limitations and Future Work

  • Profiling Overhead: The current system requires an initial profiling phase (running about 50 iterations) to gather statistics on layer sensitivity and execution costs. The authors suggest moving toward more "lightweight" indicators to reduce this setup time.
  • Scope of Parallelism: While highly effective for data-parallel training, the paper primarily focuses on this mode. Extending the "quantization-minimized" logic to tensor-parallel or pipeline-parallel setups remains an open area for research.
  • Quantization by Floor: The authors briefly mention a surprising finding: replacing complex stochastic quantization with simple "flooring" might also restore training quality, a direction they intend to investigate further.

Summary updated

Abstract

A number of production deep learning clusters have attempted to explore inference hardware for DNN training, at the off-peak serving hours with many inference GPUs idling. Conducting DNN training with a combination of heterogeneous training and inference GPUs, known as hybrid device training, presents considerable challenges due to disparities in compute capability and significant differences in memory capacity. We propose QSync, a training system that enables efficient synchronous data-parallel DNN training over hybrid devices by strategically exploiting quantized operators. According to each device’s available resource capacity, QSync selects a quantization-minimized setting for operators in the distributed DNN training graph, minimizing model accuracy degradation but keeping the training efficiency brought by quantization. We carefully design a predictor with a bi-directional mixed-precision indicator to reflect the sensitivity of DNN layers on fixed-point and floating-point low-precision operators, a replayer with a neighborhood-aware cost mapper to accurately estimate the latency of distributed hybrid mixed-precision training, and then an allocator that efficiently synchronizes workers with minimized model accuracy degradation. QSync bridges the computational graph on PyTorch to an optimized backend for quantization kernel performance and flexible support for various GPU architectures. Extensive experiments show that QSync’s predictor can accurately simulate distributed mixed-precision training with < 5% error, with a consistent 0.27 − 1.03% accuracy improvement over the from-scratch training tasks compared to uniform precision.
BibTeX
@inproceedings{zhao2024qsync,
  author = {Juntao Zhao and Borui Wan and Yanghua Peng and Haibin Lin and Yibo Zhu and Chuan Wu},
  title = {{QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices}},
  booktitle = {IPDPS},
  year = {2024},
  url = {https://openreview.net/forum?id=MYfkkjzsDY},
}

Loading related content...

Figure