Learning to Minimize the Remainder in Supervised Learning

Yan Luo, Yongkang Wong, Mohan S. Kankanhalli, Qi Zhao

TL;DR

This paper introduces Gradient Adjustment Learning (GAL), a model-agnostic approach that improves neural network training by accounting for the "remainder" terms usually ignored in first-order Taylor expansions. By using a small auxiliary module to learn and apply gradient adjustments, the authors demonstrate consistent performance gains across image classification, object detection, and regression tasks without adding any overhead to inference.

What Problem Does This Paper Address?

In standard deep learning, we update model parameters using gradient descent. Mathematically, this process relies on a first-order Taylor expansion: we approximate the loss function locally as a linear surface and move in the direction of the steepest descent.

However, Taylor's theorem includes a remainder—a collection of higher-order terms (curvatures) that we typically discard for simplicity. In high-dimensional, complex data like images, this remainder is rarely zero. Ignoring it means our "vanilla" gradients might not be the most efficient path to a local minimum. The authors argue that this remainder is actually informative and that learning to minimize it can lead to better generalization and faster convergence.

Methods

The authors propose Gradient Adjustment Learning (GAL), which consists of three main components:

  1. The Adjustment Module: A small, lightweight Multi-Layer Perceptron (MLP) that takes the model's discriminative features as input and predicts a "gradient adjustment vector."
  2. Normalization and Scaling: To ensure the adjustment doesn't overwhelm the actual gradient and cause instability, the predicted vector is normalized and scaled by a hyperparameter $\alpha$ relative to the magnitude of the vanilla gradient.
  3. Conditional Update Policy (The Safeguard): Before applying an update, the system calculates a "tentative loss." If the adjusted gradient doesn't actually lead to a lower loss than the vanilla gradient, the system falls back to the standard update. This prevents the adjustment module from "ruining" the training process during early stages.

The adjustment module itself is trained to minimize the absolute value of the remainder derived from the Taylor expansion, effectively learning the "shape" of the loss landscape that the first-order gradient misses.

Datasets

The researchers evaluated GAL across a diverse set of benchmarks:

  • Image Classification: CIFAR-10, CIFAR-100, and the large-scale ImageNet dataset.
  • Object Detection: MS COCO 2017 (using the DETR transformer-based model).
  • Regression: Three classic datasets: Boston housing, Diabetes, and California housing.

Key Insights and Ideas

  • The Remainder as Signal: Instead of treating higher-order terms as noise, GAL treats them as a supervised learning problem. By minimizing the remainder, the model finds a "tighter" approximation of the true loss surface.
  • Geometric Shortcuts: Visualization of the loss landscape shows that GAL-adjusted paths often find "shortcuts" to the local minimum, bypassing the zig-zagging behavior sometimes seen in standard optimizers like SGD or Adam.
  • Optimizer Agnostic: GAL isn't a new optimizer; it's a way to modify gradients before they reach the optimizer. The authors proved it works effectively with SGD, Lookahead, Adabound, and RMSProp.
  • Tighter Error Bounds: The paper provides a new theoretical proof (Theorem IV.4) showing that properly adjusting the gradient direction leads to a tighter remainder error bound than the conventional mathematical limits.

Key Results

GAL consistently outperformed "vanilla" training across all tested tasks:

  • Classification: On CIFAR-100, EfficientNet-B1 error dropped from 11.81% to 11.37%. On ImageNet, Top-1 accuracy for EfficientNet-B2 improved from 77.93% to 78.10%.
  • Object Detection: Using the DETR model on MS COCO, the Average Precision (AP) improved from 39.13 to 39.61.
  • Regression: The improvements were most dramatic here. For the Boston housing dataset, the Mean Absolute Error (MAE) dropped significantly from 3.95 to 2.81.
  • Efficiency: The training overhead is minimal (adding only about 11–15 milliseconds per image), and because the adjustment module is only used during training, there is zero impact on inference speed or model size for deployment.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of GAL is somewhat sensitive to the scaling factor $\beta$, which appears to be optimizer-dependent (e.g., SGD preferred $\beta=1$ while Lookahead performed better with $\beta=5$).
  • Module Architecture: While the authors found that simple MLPs outperformed CNNs for the adjustment module (likely because features are already compressed), they suggest that more sophisticated "subtle characteristic" capturing architectures could be explored.
  • Stochastic Noise: While GAL helps minimize the remainder, the underlying stochastic nature of mini-batch training still introduces noise that the module must contend with. Future work could look at how GAL interacts with different noise distributions in the training data.

Summary updated

Abstract

The learning process of deep learning methods usually updates the model’s parameters in multiple iterations. Each iteration can be viewed as the first-order approximation of Taylor’s series expansion. The remainder, which consists of higher-order terms, is usually ignored in the learning process for simplicity. This learning scheme empowers various multimedia-based applications, such as image retrieval, recommendation system, and video search. Generally, multimedia data ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> images) are semantics-rich and high-dimensional, hence the remainders of approximations are possibly non-zero. In this work, we consider that the remainder is informative and study how it affects the learning process. To this end, we propose a new learning approach, namely gradient adjustment learning (GAL), to leverage the knowledge learned from the past training iterations to adjust vanilla gradients, such that the remainders are minimized and the approximations are improved. The proposed GAL is model- and optimizer-agnostic, and is easy to adapt to the standard learning framework. It is evaluated on three tasks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> image classification, object detection, and regression, with state-of-the-art models and optimizers. The experiments show that the proposed GAL consistently enhances the evaluated models, whereas the ablation studies validate various aspects of the proposed GAL. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/luoyan407/gradient_adjustment.git</uri> .
BibTeX
@article{luo2023learning,
  author = {Yan Luo and Yongkang Wong and Mohan S. Kankanhalli and Qi Zhao},
  title = {{Learning to Minimize the Remainder in Supervised Learning}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3158066},
}
Figure