Neither Quick Nor Proper -- Evaluation of QuickProp for Learning Deep Neural Networks

Clemens-Alexander Brust, Sven Sickert, Marcel Simon, Erik Rodner, Joachim Denzler

TL;DR

Researchers revisited QuickProp, a classic second-order optimization algorithm from the late 1980s, to see if it could accelerate the training of modern deep Convolutional Neural Networks (CNNs). While QuickProp showed a strange knack for generalization on tiny artificial datasets, it ultimately failed to compete with standard Gradient Descent (GD) on complex, real-world semantic segmentation tasks.

What Problem Does This Paper Address?

Training deep neural networks is computationally expensive and time-consuming. While first-order methods like Gradient Descent (and its variants like Adam or SGD with momentum) are the industry standard, they only use the slope (gradient) of the error surface to find the minimum.

Second-order methods, which look at the curvature (the Hessian matrix) of the error surface, theoretically offer a more direct path to the optimum. However, calculating the full Hessian for a model with millions of parameters is impossible. The authors wanted to know if QuickProp—a "Quasi-Newton" method that uses a cheap approximation of this curvature—could provide a shortcut for modern computer vision tasks like semantic segmentation.

Methods

The researchers implemented the original QuickProp algorithm and compared it against standard Gradient Descent.

  • QuickProp Logic: The algorithm assumes the error surface for each weight is a simple parabola. By looking at the current gradient and the previous gradient, it tries to jump directly to the bottom of that parabola.
  • The Update Rule: Unlike the additive updates of momentum-based GD, QuickProp uses a multiplicative update based on the difference between gradients.
  • Heuristics: To prevent the algorithm from making "infinite" jumps or going the wrong way, it uses a "maximum growth factor" ($\mu$) to cap how much a weight can change in one step.
  • Architecture: They applied these optimizers to Fully Convolutional Networks (FCNs), which are designed to classify every single pixel in an image (semantic segmentation).
Fig. 1. Architecture of convolutional neural networks used for semantic segmentation.

Datasets

The study utilized two distinct datasets to test how the algorithm scales with complexity:

  1. T.O.Y. Dataset: A tiny, artificial dataset consisting of a single image with three classes. This allowed the team to run 200 repetitions to ensure statistical significance.
  2. LabelMeFacade: A real-world dataset of 945 urban scene images (buildings, cars, trees, etc.). This provided a much more rigorous test of the algorithm's ability to handle visual variety and high-dimensional parameter spaces.
Fig. 2. Datasets used in our experiments: A simple three class scenario in the T.O.Y. dataset (left) and a multi-class setting of urban scenes in the LabelMeFacade dataset [5] (right). Depicted are both input image and the corresponding ground truth labeling as color coding.

Key Insights and Ideas

  • The Generalization Paradox: In the simple T.O.Y. experiments, QuickProp actually had higher training loss than Gradient Descent but lower testing loss. This suggests that, in very specific low-complexity scenarios, QuickProp might have better generalization properties.
  • Complexity is the Enemy: As the researchers scaled up the number of filters and layers in the network, QuickProp’s performance steadily degraded compared to GD. The gap in performance widened as the models became more "modern" in size.
  • Numerical Instability: The authors note that modern deep networks involve long chains of backpropagation. This leads to higher numerical errors in gradient calculation, which wreaks havoc on QuickProp’s finite-difference approximation of the second derivative.

Key Results

  • Accuracy Gap: On the LabelMeFacade dataset, QuickProp's accuracy was consistently about 5 percentage points lower than Gradient Descent.
  • Class-wise Performance: On the toy dataset, while both reached ~78% overall accuracy, QuickProp’s mean class-wise accuracy (41%) was significantly worse than GD’s (62%), indicating it struggles to balance learning across different categories.
  • Scaling Trends: Experiments scaling the number of filters (parameter $k$) showed that QuickProp performs better on smaller networks, but its training loss rises much faster than GD's as the network grows.

Limitations and Future Work

The primary limitation is QuickProp's core assumption: that weights are independent and the error surface is locally quadratic. In deep CNNs, weights are highly interdependent, and the error landscape is notoriously non-convex and complex. Furthermore, the numerical instability of gradient differences in deep architectures makes the "Quick" part of QuickProp quite unreliable.

The authors conclude that while second-order methods are a promising area for research, the classic QuickProp algorithm cannot be recommended for contemporary deep learning tasks without significant modifications to handle the scale and numerical sensitivities of modern architectures.

Summary updated

Abstract

Neural networks and especially convolutional neural networks are of great interest in current computer vision research. However, many techniques, extensions, and modifications have been published in the past, which are not yet used by current approaches. In this paper, we study the application of a method called QuickProp for training of deep neural networks. In particular, we apply QuickProp during learning and testing of fully convolutional networks for the task of semantic segmentation. We compare QuickProp empirically with gradient descent, which is the current standard method. Experiments suggest that QuickProp can not compete with standard gradient descent techniques for complex computer vision tasks like semantic segmentation.
BibTeX
@misc{brust2016neither,
  author = {Clemens-Alexander Brust and Sven Sickert and Marcel Simon and Erik Rodner and Joachim Denzler},
  title = {{Neither Quick Nor Proper -- Evaluation of QuickProp for Learning Deep Neural Networks}},
  year = {2016},
  eprint = {1606.04333},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.04333},
}
Figure