Resilient Backpropagation (Rprop) for Batch-learning in TensorFlow

Ciprian Florescu, Christian Igel

TL;DR

Training neural networks often involves a time-consuming process of hyperparameter tuning. This paper introduces a TensorFlow implementation of Resilient Backpropagation (Rprop), a classic optimization algorithm that effectively eliminates hyperparameter tuning for batch learning. The authors demonstrate that Rprop, using only its default settings, can outperform "fully tuned" modern heavyweights like Adam and RMSprop when training Recurrent Neural Networks (RNNs).

The Challenge of Hyperparameter Tuning

In the modern deep learning era, mini-batch stochastic gradient descent has become the standard. While mini-batches are essential for massive datasets, they introduce a significant headache: hyperparameter tuning. Finding the optimal learning rate, decay, and momentum for algorithms like Adam or RMSprop can sometimes take more time than the actual training process.

However, there are many scenarios—such as rapid prototyping, working with smaller datasets, or training brittle architectures like RNNs—where batch learning (using the entire dataset for every update) remains highly viable. The challenge is that standard deep learning frameworks like TensorFlow were not originally built to support Rprop’s unique requirements, such as accessing both current and past gradients and loss values to decide whether to "backtrack" a weight update.

Methods: Implementing iRprop+

The authors implemented four variants of Rprop in TensorFlow, focusing primarily on iRprop+. Unlike standard gradient descent, which uses the magnitude of the gradient to determine the step size, Rprop relies exclusively on the sign of the gradient:

  1. Directional Updates: If the gradient sign stays the same across iterations, the step size increases, accelerating the optimizer toward the minimum.
  2. The "Overshoot" Correction: If the gradient sign flips, it indicates the optimizer jumped over a local minimum. The step size is subsequently decreased.
  3. Weight Backtracking (The '+' in iRprop+): If a step actually makes the total error worse, iRprop+ undoes the last update and resets the gradient tracking.

Crucially, the authors extended the standard TensorFlow optimizer interface to allow the algorithm to "look back" at the objective function's value, a necessity for the backtracking logic.

Experiments and Datasets

The researchers evaluated their implementation on two RNN-based tasks:

  • Human Activity Recognition (HAR): A movement classification task using a dataset of motion sensor signals from smartphones. The goal was to classify activities (e.g., walking, sitting) using a 2-layer RNN with Gated Recurrent Units (GRUs).
  • Lorenz Time Series Modeling: A regression task predicting the future dynamics of the chaotic Lorenz attractor, utilizing a 1-layer RNN with 42 GRUs and a squared-error loss function.

Key Insights and Results

Training loss comparisons for Movement Classification and Lorenz Time Series Modeling.
  • The "No-Tuning" Advantage: For batch learning, Rprop's default hyperparameters (established in the 1990s) proved remarkably robust. In every test, the "out-of-the-box" Rprop beat other optimizers that had been meticulously tuned via grid search.
  • Superior Performance: On both the HAR classification and Lorenz regression tasks, iRprop+ with default parameters achieved lower training errors than Adam and RMSprop, even after those algorithms were "fully tuned" across hundreds of parameter combinations.
  • Convergence Dynamics: While tuned mini-batch Adam learned faster in the very early epochs of the HAR task, iRprop+ eventually overtook it, reaching a better final solution after 50 epochs.
  • Batch Speed: On specific hardware (NVIDIA Tesla K80), the authors found batch learning for RNNs was 15 times faster than mini-batching. Mini-batching creates a bottleneck when aggregating gradients for multi-layer RNNs, leading to poor GPU utilization.
  • Overcoming Vanishing Gradients: By ignoring the absolute value of the gradient and focusing only on the sign, Rprop naturally avoids the "vanishing gradient" problem often found in deep networks and RNNs, where gradients become too small to trigger meaningful updates.

Limitations and Future Work

The primary limitation is that Rprop is strictly a batch learning algorithm. The authors acknowledge that it does not perform well in mini-batch scenarios, which are the standard for massive "Big Data" applications. While some mini-batch variants of Rprop exist, they were not the focus of this implementation.

Ultimately, the authors suggest that for researchers working on RNNs or smaller-scale scientific modeling where batch learning is feasible, iRprop+ should be the "method of choice" to save hours of fiddling with learning rates. Future work will likely involve optimizing the TensorFlow C++ kernels further for even tighter integration into the ecosystem.

Summary updated

BibTeX
@inproceedings{florescu2018resilient,
  author = {Ciprian Florescu and Christian Igel},
  title = {{Resilient Backpropagation (Rprop) for Batch-learning in TensorFlow}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
}
Figure