signADAM: Learning Confidences for Deep Neural Networks

Dong Wang, Yicheng Liu, Wenwo Tang, Fanhua Shang, Hongying Liu, Qigong Sun, Licheng Jiao

Introduction

In the high-stakes world of deep learning optimization, the battle has long been between the reliability of Stochastic Gradient Descent (SGD) and the speed of adaptive methods like ADAM. While ADAM is the go-to for many, it isn't perfect—it can be computationally heavy and sometimes struggles with generalization.

A new paper, "signADAM: Learning Confidences for Deep Neural Networks," introduces a clever hybrid approach. By combining the simplicity of sign-based gradients with a biologically inspired "confidence function," the authors propose signADAM++, an optimizer that doesn't just move faster—it moves smarter.

Problem Statement

Training Deep Neural Networks (DNNs) is essentially a massive non-convex optimization puzzle. Two major challenges persist:

  1. The Vanishing Gradient Problem: In high-dimensional spaces, gradients can become infinitesimally small, stalling training.
  2. Unbalanced Feature Learning: Standard optimizers often fit "easy" features too quickly, leading the model to settle into "sharp minima"—regions of the loss landscape that look good during training but fail to generalize to new data.

Existing solutions like SIGN-SGD (which only looks at whether a gradient is positive or negative) are efficient but "blind" to the relative importance of different gradient components. The authors sought a way to distinguish between useful signal and useless noise during the update step.

Methodology: Learning with Confidence

The authors propose two main algorithms:

  • signADAM: This version takes the core ADAM logic but replaces the raw stochastic gradient with its sign ($\pm 1$). This reduces the impact of extreme outliers and simplifies the communication overhead in distributed settings.
  • signADAM++: This is the "premium" version. It introduces a Confidence Function. If the magnitude of a gradient component is below a certain threshold ($\alpha$), the optimizer treats it as "low confidence" and sets it to zero. If it’s above the threshold, it uses the sign.
The signADAM++ calculation unit processes gradients through a confidence function to filter out low-confidence signals.

Why does this work? It creates gradient sparsity. By ignoring small updates, the model focuses its "attention" on the most informative samples (often the ones the model is currently getting wrong). This is conceptually similar to Online Hard Example Mining (OHEM) but baked directly into the optimizer.

Key Insights

  • Biological Inspiration: The authors point to neuroscience research showing that only 1-4% of neurons in the brain are active at any given time. signADAM++ mimics this "sparse coding" by only updating weights when the gradient signal is strong enough.
  • The Loss Landscape Connection: The paper argues that by treating features more equally and ignoring "easy" noise, the optimizer avoids falling into sharp, narrow valleys in the loss landscape. Instead, it tends toward "flat minima," which are known to generalize much better to test data.
  • Sign > Magnitude: In many noisy mini-batch scenarios, the direction of the gradient is a more stable signal than its exact magnitude. signADAM++ leans into this stability.

Empirical Evaluation

The researchers put their algorithms to the test on several standard benchmarks including MNIST (LeNet, LSTM), CIFAR-10 (VGG-19, ResNet-18), and CIFAR-100 (VGG-19).

The empirical results are impressive, particularly regarding speed and convergence:

  • Faster Convergence: On MNIST (LeNet), signADAM++ reached the target error rate in just 20 epochs, compared to 58 epochs for standard ADAM.
  • Superior Accuracy: On the challenging CIFAR-100 dataset, a refined version of the algorithm (signADAM++-AW) achieved 72.31% accuracy, significantly outperforming ADAMW (69.11%) and standard ADAM (68.62%).
Performance comparison on VGG19 shows signADAM++ achieving lower training loss and test error compared to standard ADAM and SIGN-based methods.

Limitations and Future Work

While signADAM++ shows great promise, the authors acknowledge some hurdles:

  • Hyperparameter Tuning: The performance depends on the confidence threshold ($\alpha$). If ($\alpha$) is too high, the model might stop learning specific features entirely.
  • Data Sensitivity: On CIFAR-100, the basic signADAM++ initially performed slightly worse than ADAM because the lower number of samples per class made the "zeroing out" of gradients too aggressive. The authors had to develop an Adaptive Weight-decay (AW) technique—using a dynamic, standard-deviation-based threshold—to solve this.

The authors suggest that future work could involve further refining these adaptive thresholds to make the optimizer "plug-and-play" across even more diverse datasets without manual tuning.

Summary updated

Abstract

In this paper, we propose a new first-order gradient-based algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in sign-based methods, e.g., SIGN-SGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we define a confidence function to distinguish different components of gradients and apply it to our algorithm. It can generate more sparse gradients than existing algorithms do. We call this new algorithm signADAM++. In particular, both our algorithms are easy to implement and can speed up training of various deep neural networks. The motivation of signADAM++ is preferably learning features from the most different samples by updating large and useful gradients regardless of useless information in stochastic gradients. We also establish theoretical convergence guarantees for our algorithms. Empirical results on various datasets and models show that our algorithms yield much better performance than many state-of-the-art algorithms including SIGN-SGD, SIGNUM and ADAM. We also analyze the performance from multiple perspectives including the loss landscape and develop an adaptive method to further improve generalization. The source code is available at https://github.com/DongWanginxdu/signADAM-Learn-by-Confidence.
BibTeX
@inproceedings{wang2019signadam,
  author = {Dong Wang and Yicheng Liu and Wenwo Tang and Fanhua Shang and Hongying Liu and Qigong Sun and Licheng Jiao},
  title = {{signADAM: Learning Confidences for Deep Neural Networks}},
  booktitle = {Workshops @ ICDM},
  year = {2019},
  doi = {10.1109/icdmw.2019.00037},
}
Figure