ADMMiRNN: Training RNN with Stable Convergence via An Efficient ADMM Approach

Yu Tang, Zhigang Kan, Dequan Sun, Linbo Qiao, Jingjing Xiao, Zhiquan Lai, Dongsheng Li

Introduction

Recurrent Neural Networks (RNNs) are a cornerstone of sequence modeling, yet they are notoriously difficult to train. The standard approach, Backpropagation Through Time (BPTT), often suffers from vanishing and exploding gradients because weights are reused across every timestep. Furthermore, RNNs are highly sensitive to weight and bias initialization; a poor starting point can prevent convergence entirely.

While gradient-based optimizers like SGD, Adam, and RMSProp are standard, they remain susceptible to these fundamental issues. This paper introduces ADMMiRNN, a novel framework that leverages the Alternating Direction Method of Multipliers (ADMM) to train RNNs, offering a robust, gradient-free alternative with stable convergence.

Methodology

The ADMMiRNN framework transforms the RNN training process into a constrained optimization problem that can be solved efficiently without traditional backpropagation.

Unfolding and Decoupling

The core of ADMMiRNN lies in unfolding the RNN into a sequence of $N$ continuous cells. By treating the recurrent units as a sequence, the global optimization problem is broken down into eight smaller, manageable sub-problems. This allows for the separate updating of weights ($u, w, b$) and states ($a, s, o$), effectively decoupling the parameters that usually cause gradient issues.

The structure of a standard RNN cell (a) and its unfolded representation (b) used in ADMMiRNN.

Efficient Optimization

To ensure the method is computationally viable, the authors employ a Linearized Proximal Mapping technique. This avoids the expensive matrix inversions (pseudo-inverses) typically required in vanilla ADMM updates, significantly accelerating the training process. Additionally, specific mathematical approximations, such as linearizing the tanh activation for small inputs, are used to simplify calculations without compromising accuracy.

Parallelization Strategies

To further enhance performance, the authors developed parallel versions of the framework:

  • Synchronous Parallel (SP-ADMMiRNN): Workers synchronize at each iteration.
  • Asynchronous Parallel (AP-ADMMiRNN): Workers update the master node independently. This version is particularly effective as it eliminates idle time caused by slower workers ("stragglers").
Comparison of Synchronous (a) and Asynchronous (b) parallel training schemes for ADMMiRNN.

Experimental Evaluation

The researchers evaluated ADMMiRNN across three benchmarks: MNIST (sequential pixel classification), IMDb (sentiment analysis), and a general Text Classification task.

Key Results

  • Convergence Stability: ADMMiRNN demonstrated remarkable immunity to poor initialization. In repeated tests, it showed significantly lower standard deviation in training and test loss compared to SGD, Adam, and RMSProp.
  • Sequence Length Robustness: Unlike BPTT-based models, which degrade as sequence length increases, ADMMiRNN maintained stable performance regardless of input length.
  • Training Speed: On MNIST and IMDb, the framework reached peak accuracy in fewer iterations than gradient-based methods.
  • Parallel Efficiency: The AP-ADMMiRNN variant achieved near-linear speedup. With 3 workers, it completed 50 iterations in 92.64 seconds, compared to 266.53 seconds for the non-parallel version.

Conclusion and Future Work

ADMMiRNN provides a stable and efficient alternative to gradient-based RNN training, effectively bypassing the vanishing and exploding gradient problems. While current results on medium-scale datasets are promising, future work will focus on scaling the framework to larger datasets and extending the decomposition logic to more complex architectures like LSTMs and GRUs.

Summary updated

Abstract

It is hard to train Recurrent Neural Network (RNN) with stable convergence and avoid gradient vanishing and exploding, as the weights in the recurrent unit are repeated from iteration to iteration. Moreover, RNN is sensitive to the initialization of weights and bias, which brings difficulty in the training phase. With the gradient-free feature and immunity to unsatisfactory conditions, the Alternating Direction Method of Multipliers (ADMM) has become a promising algorithm to train neural networks beyond traditional stochastic gradient algorithms. However, ADMM could not be applied to train RNN directly since the state in the recurrent unit is repetitively updated over timesteps. Therefore, this work builds a new framework named ADMMiRNN upon the unfolded form of RNN to address the above challenges simultaneously and provides novel update rules and theoretical convergence analysis. We explicitly specify essential update rules in the iterations of ADMMiRNN with deliberately constructed approximation techniques and solutions to each sub-problem instead of vanilla ADMM. Numerical experiments are conducted on MNIST and text classification tasks, where ADMMiRNN achieves convergent results and outperforms compared baselines. Furthermore, ADMMiRNN trains RNN more stably without gradient vanishing or exploding compared to the stochastic gradient algorithms. Source code has been available at https://github.com/TonyTangYu/ADMMiRNN .
BibTeX
@inproceedings{tang2020admmirnn,
  author = {Yu Tang and Zhigang Kan and Dequan Sun and Linbo Qiao and Jingjing Xiao and Zhiquan Lai and Dongsheng Li},
  title = {{ADMMiRNN: Training RNN with Stable Convergence via An Efficient ADMM Approach}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67661-2\_1},
}
Figure