TL;DR
Researchers revisited QuickProp, a classic second-order optimization algorithm from the late 1980s, to see if it could accelerate the training of modern deep Convolutional Neural Networks (CNNs). While QuickProp showed a strange knack for generalization on tiny artificial datasets, it ultimately failed to compete with standard Gradient Descent (GD) on complex, real-world semantic segmentation tasks.
What Problem Does This Paper Address?
Training deep neural networks is computationally expensive and time-consuming. While first-order methods like Gradient Descent (and its variants like Adam or SGD with momentum) are the industry standard, they only use the slope (gradient) of the error surface to find the minimum.
Second-order methods, which look at the curvature (the Hessian matrix) of the error surface, theoretically offer a more direct path to the optimum. However, calculating the full Hessian for a model with millions of parameters is impossible. The authors wanted to know if QuickProp—a "Quasi-Newton" method that uses a cheap approximation of this curvature—could provide a shortcut for modern computer vision tasks like semantic segmentation.
Methods
The researchers implemented the original QuickProp algorithm and compared it against standard Gradient Descent.
- QuickProp Logic: The algorithm assumes the error surface for each weight is a simple parabola. By looking at the current gradient and the previous gradient, it tries to jump directly to the bottom of that parabola.
- The Update Rule: Unlike the additive updates of momentum-based GD, QuickProp uses a multiplicative update based on the difference between gradients.
- Heuristics: To prevent the algorithm from making "infinite" jumps or going the wrong way, it uses a "maximum growth factor" ($\mu$) to cap how much a weight can change in one step.
- Architecture: They applied these optimizers to Fully Convolutional Networks (FCNs), which are designed to classify every single pixel in an image (semantic segmentation).
Datasets
The study utilized two distinct datasets to test how the algorithm scales with complexity:
- T.O.Y. Dataset: A tiny, artificial dataset consisting of a single image with three classes. This allowed the team to run 200 repetitions to ensure statistical significance.
- LabelMeFacade: A real-world dataset of 945 urban scene images (buildings, cars, trees, etc.). This provided a much more rigorous test of the algorithm's ability to handle visual variety and high-dimensional parameter spaces.
Key Insights and Ideas
- The Generalization Paradox: In the simple T.O.Y. experiments, QuickProp actually had higher training loss than Gradient Descent but lower testing loss. This suggests that, in very specific low-complexity scenarios, QuickProp might have better generalization properties.
- Complexity is the Enemy: As the researchers scaled up the number of filters and layers in the network, QuickProp’s performance steadily degraded compared to GD. The gap in performance widened as the models became more "modern" in size.
- Numerical Instability: The authors note that modern deep networks involve long chains of backpropagation. This leads to higher numerical errors in gradient calculation, which wreaks havoc on QuickProp’s finite-difference approximation of the second derivative.
Key Results
- Accuracy Gap: On the LabelMeFacade dataset, QuickProp's accuracy was consistently about 5 percentage points lower than Gradient Descent.
- Class-wise Performance: On the toy dataset, while both reached ~78% overall accuracy, QuickProp’s mean class-wise accuracy (41%) was significantly worse than GD’s (62%), indicating it struggles to balance learning across different categories.
- Scaling Trends: Experiments scaling the number of filters (parameter $k$) showed that QuickProp performs better on smaller networks, but its training loss rises much faster than GD's as the network grows.
Limitations and Future Work
The primary limitation is QuickProp's core assumption: that weights are independent and the error surface is locally quadratic. In deep CNNs, weights are highly interdependent, and the error landscape is notoriously non-convex and complex. Furthermore, the numerical instability of gradient differences in deep architectures makes the "Quick" part of QuickProp quite unreliable.
The authors conclude that while second-order methods are a promising area for research, the classic QuickProp algorithm cannot be recommended for contemporary deep learning tasks without significant modifications to handle the scale and numerical sensitivities of modern architectures.
Summary updated