Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends

Guoxin Cui, Jiafeng Guo, Yixing Fan, Yanyan Lan, Xueqi ChengIEEE Access 2019 · DOI · OpenReview

Introduction

Asynchronous Stochastic Gradient Descent (A-SGD) is a cornerstone of large-scale machine learning, enabling high hardware utilization by allowing multiple worker nodes to update a central parameter server without synchronization. However, this lack of coordination introduces the "stale gradient" problem: updates calculated on outdated model versions introduce noise, causing parameter "jitter" and hindering convergence. While existing solutions often rely on complex second-order approximations, this paper proposes Trend-Smooth, a lightweight optimization wrapper that leverages parameter trends to dampen noisy updates.

Background: Synchronous vs. Asynchronous SGD

In Synchronous SGD (S-SGD), the server waits for all workers to complete their gradient calculations before performing a global update. This ensures consistency but creates a bottleneck known as the "straggler problem."

Synchronous SGD (S-SGD) architecture where the server aggregates gradients from all workers before updating parameters.

Conversely, A-SGD allows workers to update the server independently. While faster in terms of iterations per second, the gradients received by the server are often "stale," leading to the fluctuations that Trend-Smooth aims to mitigate.

Asynchronous SGD (A-SGD) architecture where the server updates parameters immediately upon receiving a gradient from any worker.

Methodology: The Trend-Smooth Algorithm

Trend-Smooth operates on a simple principle: updates that align with the long-term trajectory of the model are likely signal, while those that contradict it are likely noise.

1. Trend Tracking

The server maintains a history (stack) of recent parameter values. A "trend" for each parameter dimension is established by calculating the difference between the current parameter value and a historical value from the stack.

2. Gradient Filtering

Upon receiving a gradient from a worker, the server compares the sign of the gradient with the sign of the established trend.

  • Alignment: If the gradient direction matches the trend, the update is applied normally.
  • Contradiction: If the gradient moves against the trend, it is flagged as potential stale noise. The learning rate for that specific dimension is then scaled by a shrinking factor $\alpha$ (typically set to 0.8).

3. Optimizer Compatibility

Trend-Smooth is designed as a modular wrapper. It can be integrated with standard optimizers like Momentum, RMSProp, and Adam, enhancing their robustness in asynchronous environments without significant computational overhead.

Experimental Evaluation

The authors evaluated Trend-Smooth on two standard benchmarks:

  • MNIST: A 3-layer fully-connected network.
  • CIFAR-10: A 5-layer Convolutional Neural Network (CNN).

Key Findings

  • Accuracy Gains: On MNIST, Trend-Smooth achieved 98.07% accuracy, narrowing the gap between standard A-SGD (97.42%) and sequential SGD (98.26%).
  • Convergence Speed: Trend-Smooth demonstrated faster convergence in wall-clock time compared to other asynchronous baselines like Delay-Compensated ASGD (A-DC).
  • Synergy with Adaptive Methods: Integrating Trend-Smooth with Adam on CIFAR-10 improved accuracy from 80.73% to 81.92%, demonstrating its effectiveness even when paired with sophisticated optimizers.

Discussion and Conclusion

Trend-Smooth addresses the staleness problem in A-SGD by using parameter history as a global memory to maintain consistency. Its sign-based approach avoids the heavy computational costs of Hessian-based methods.

While effective for moderate-scale networks, future research is needed to assess its performance on massive architectures like Transformers and to optimize the memory overhead required for storing parameter history in high-dimensional models. Additionally, exploring adaptive mechanisms for the shrinking factor $\alpha$ could further enhance its performance.

Summary updated

Abstract

Stochastic gradient descent(SGD) is the fundamental sequential method in training large scale machine learning models. To accelerate the training process, researchers proposed to use the asynchronous stochastic gradient descent (A-SGD) method in model learning. However, due to the stale information when updating parameters, A-SGD converges more slowly than SGD in the same iteration number. Moreover, A-SGD often converges to a high loss value and results in lower model accuracy. In this paper, we propose a novel algorithm called Trend-Smooth which can be adapted to the asynchronous parallel environment to overcome the above problems. Specifically, Trend-Smooth makes use of the parameter trend during the training process to shrink the learning rate of some dimensions where the gradients' directions are opposite to the trends of parameters. Experiments on MNIST and CIFAR-10 datasets confirm that Trend-Smooth can accelerate the convergence speed in asynchronous training process. The test accuracy that Trend-Smooth achieves is shown to be higher than other asynchronous parallel baseline methods, and is very close to the SGD method. Moreover, Trend-Smooth can also be combined with other adaptive learning rate methods(like Momentum, RMSProp and Adam) in the asynchronous parallel environment to promote their performance.
BibTeX
@article{cui2019trend,
  author = {Guoxin Cui and Jiafeng Guo and Yixing Fan and Yanyan Lan and Xueqi Cheng},
  title = {{Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2949611},
}
Figure