Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems

Jihyun Lim, Junhyuk Jo, Chanhyeok Ko, Young Min Go, Jimin Hwa, Sunwoo LeearXiv 2025 · DOI · arXiv

Introduction

Parallel deep learning typically operates under the assumption of hardware homogeneity, where all computing units (workers) possess identical processing capabilities. However, modern production environments often utilize heterogeneous systems, combining high-performance GPUs with relatively slower CPUs. Standard parallelization strategies, such as Synchronous Stochastic Gradient Descent (SGD), suffer significantly in these environments due to the "straggler problem," where faster units must idle while waiting for slower ones to complete their tasks.

This paper introduces Biased Local SGD, a novel training framework designed to eliminate synchronization overhead by intentionally introducing controlled bias into the training process. By leveraging unbalanced local updates and biased aggregation, the framework achieves up to a 32x speedup in training time while maintaining or improving model accuracy.

The Challenge of Hardware Heterogeneity

In a typical mixed CPU-GPU setup, Synchronous SGD forces expensive GPU resources to remain idle for a substantial portion of the training cycle. Even "Local SGD," which reduces communication frequency by allowing workers to perform multiple local updates before averaging, still requires all workers to synchronize at every aggregation point. This results in significant "blocking time," where the system's overall throughput is dictated by its slowest component.

Timing breakdown across different datasets showing significant GPU idle time (blocking time) due to synchronization with slower CPUs.

The Biased Local SGD Framework

To address these inefficiencies, the authors propose a System-aware Unbalanced Local SGD framework. Instead of enforcing a uniform workload, the framework allows faster workers to perform a higher number of local updates than slower workers within the same time window. To ensure convergence and maintain model quality despite this imbalance, the framework injects two forms of controlled bias:

  1. Bias via Loss-based Data Sampling: Fast workers are prioritized with "harder" data samples—those exhibiting higher loss values. This ensures that the most capable hardware focuses on the most informative data, accelerating the learning trajectory.
  2. Bias via Model Aggregation: During the global model update, contributions from faster workers are assigned higher weights. Because these workers have performed more updates on more critical data, their local models are considered more representative of the target distribution.
Schematic of the Biased Local SGD framework showing the workflow of unbalanced updates, loss-based sampling for fast resources, and weighted model aggregation.

Crucially, the authors provide theoretical proof that this biased approach guarantees convergence for smooth, non-convex optimization problems, effectively turning "bias" from a statistical hurdle into a system-level optimization tool.

Experimental Evaluation

The framework was evaluated across a diverse range of benchmarks, including computer vision (CIFAR-10, CIFAR-100, Oxford Flowers 102, Tiny ImageNet) and natural language processing (AG News).

Key Results

  • Dramatic Acceleration: For ResNet20 on CIFAR-10, Biased Local SGD achieved a 32x speedup over synchronous SGD.
  • Wall-clock Efficiency: When measured against real-world time rather than epoch counts, the proposed method reached target accuracy levels significantly faster than both Synchronous SGD and standard "Balanced" Local SGD.
  • Effective Resource Utilization: The study demonstrated that adding "slow" CPUs to a GPU-only cluster actually improved final accuracy within a fixed time budget, proving the framework successfully extracts value from hardware that would otherwise be considered a liability.

Discussion and Future Work

Biased Local SGD represents a shift in distributed optimization philosophy, prioritizing system efficiency over strict statistical unbiasedness. By setting local update counts based on a "performance gap ratio" ($\gamma$), the framework harmonizes hardware with vastly different performance profiles.

While highly effective, the approach does introduce minor overheads, such as the memory required to store loss values for sampling (approximately 390 KB for 100,000 samples) and the computational cost of sorting samples by loss. Future research will focus on extending this single-node multi-resource framework to multi-node distributed environments to further scale its efficiency gains.

Summary updated

BibTeX
@misc{lim2025biased,
  author = {Jihyun Lim and Junhyuk Jo and Chanhyeok Ko and Young Min Go and Jimin Hwa and Sunwoo Lee},
  title = {{Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems}},
  year = {2025},
  eprint = {2508.08540},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.08540},
}
Figure