Evaluation of QuickProp for Learning Deep Neural Networks -- A Critical Review

Clemens-Alexander Brust, Sven Sickert, Marcel Simon, Erik Rodner, Joachim Denzler

Abstract

Neural networks and especially convolutional neural networks are of great interest in current computer vision research. However, many techniques, extensions, and modifications have been published in the past, which are not yet used by current approaches. In this paper, we study the application of a method called QuickProp for training of deep neural networks. In particular, we apply QuickProp during learning and testing of fully convolutional networks for the task of semantic segmentation. We compare QuickProp empirically with gradient descent, which is the current standard method. Experiments suggest that QuickProp can not compete with standard gradient descent techniques for complex computer vision tasks like semantic segmentation.

Introduction

In the rapidly evolving field of deep learning, researchers often prioritize novel architectures over revisiting historical optimization algorithms. This study re-evaluates QuickProp, a second-order optimization method originally proposed by Scott Fahlman in 1988. While modern deep learning predominantly utilizes first-order methods like Stochastic Gradient Descent (SGD) or Adam, this research investigates whether QuickProp's quasi-Newton approach can offer advantages in the demanding task of semantic segmentation using deep Convolutional Neural Networks (CNNs).

Methodology: QuickProp and FCNs

QuickProp is a Quasi-Newton method designed to accelerate convergence by approximating the error surface as a simple parabola for each weight. It assumes that weights can be optimized independently, using the change in gradients between successive steps to estimate local curvature and "jump" to the predicted minimum.

Fully Convolutional Network (FCN) architecture used for semantic segmentation tasks.

The authors implemented QuickProp according to its original specifications, including a "maximum growth factor" to prevent unstable steps and specific handling for gradient sign changes. This was tested against standard Gradient Descent (GD) using Fully Convolutional Networks (FCNs), which are designed to produce pixel-wise label estimations through a series of convolutional, pooling, and activation layers.

Experimental Setup and Datasets

To evaluate the scalability of QuickProp, the study utilized two distinct benchmarks:

  1. T.O.Y. Dataset: A synthetic, low-dimensional dataset featuring a single image with three classes. This allowed for extensive repetitions to establish statistical significance.
  2. LabelMeFacade: A complex, real-world dataset consisting of 945 urban scene images with 8 semantic classes, representing a high-dimensional optimization challenge.
Comparison of the synthetic T.O.Y. dataset (left) and real-world urban scene images with their corresponding ground truth segmentations.

Key Insights: The Scaling Challenge

The experiments revealed several critical weaknesses in applying QuickProp to modern deep architectures:

  • The Independence Fallacy: QuickProp’s fundamental assumption of weight independence fails in deep CNNs, where weights across layers are highly co-dependent. This leads to inefficient optimization as network depth increases.
  • Numerical Fragility: The algorithm's reliance on the difference between gradients makes it highly sensitive to the numerical noise inherent in deep backpropagation.
  • The Generalization Paradox: Interestingly, on the T.O.Y. dataset, QuickProp achieved lower test loss than GD despite higher training loss, suggesting a potential bias toward flatter minima. However, this benefit did not translate to larger datasets.

Results and Conclusion

On the LabelMeFacade dataset, QuickProp's accuracy consistently lagged behind Gradient Descent by approximately 5 percentage points. Furthermore, on the toy dataset, while QuickProp had decent overall accuracy, its mean class-wise accuracy was only 41%, compared to 62% for Gradient Descent. As the number of filters (network width) or layers (depth) increased, the performance gap widened significantly.

The study concludes that while second-order information remains valuable, the original QuickProp algorithm is ill-suited for the non-convex landscapes of modern deep learning. Its quadratic assumptions are too simplistic for high-parameter models, and standard gradient descent techniques remain the superior choice for complex computer vision tasks.

Summary updated

Abstract

Neural networks and especially convolutional neural networks are of great interest in current computer vision research. However, many techniques, extensions, and modifications have been published in the past, which are not yet used by current approaches. In this paper, we study the application of a method called QuickProp for training of deep neural networks. In particular, we apply QuickProp during learning and testing of fully convolutional networks for the task of semantic segmentation. We compare QuickProp empirically with gradient descent, which is the current standard method. Experiments suggest that QuickProp can not compete with standard gradient descent techniques for complex computer vision tasks like semantic segmentation.
BibTeX
@misc{brust2016evaluation,
  author = {Clemens-Alexander Brust and Sven Sickert and Marcel Simon and Erik Rodner and Joachim Denzler},
  title = {{Evaluation of QuickProp for Learning Deep Neural Networks -- A Critical Review}},
  year = {2016},
  eprint = {1606.04333},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.04333},
}
Figure