Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks

Leni Ven, Johannes LedererarXiv 2021 · OpenReview

TL;DR

This paper argues that the vanishing gradient problem, which has led to the decline of sigmoid-type activation functions like tanh, can be effectively mitigated. The authors demonstrate that two simple techniques, L2 regularization and a rescaling trick, can revive the use of sigmoid activations by preventing gradients from becoming too small during training.

What Problem Does This Paper Address?

Deep learning models rely on gradient-based optimization algorithms to learn. A critical issue that can hinder this learning process is the "vanishing gradient problem," where gradients become extremely small, causing the model's parameters to update very slowly or not at all. This often leads to models getting stuck in suboptimal configurations. Historically, this problem has been a major reason for the shift away from sigmoid-type activation functions (like tanh) towards piecewise-linear functions (like ReLU), which are less prone to vanishing gradients. This paper revisits this problem specifically for sigmoid-type activations, aiming to show they can still be viable options with the right techniques.

Methods

The authors delve into the mathematical underpinnings of the vanishing gradient problem in sigmoid-type networks. They identify two primary sources:

  1. Large Parameters: When individual weights in the network become very large, the output of the sigmoid function approaches its saturation points (-1 or 1), leading to near-zero derivatives.
  2. Many Layers: In deep networks, the cumulative effect of many layers, even with moderate parameters, can cause gradients to shrink exponentially as they propagate backward.

To combat these issues, they propose two remedies:

  • Regularization (for Large Parameters): They show that standard L2 regularization (weight decay) can effectively counteract the vanishing gradients caused by large individual parameters. The regularization term adds a penalty proportional to the square of the weights, which, when combined with the gradient of the loss, prevents the overall gradient from becoming too small.
  • Rescaling (for Many Layers): For the many-layers problem, they introduce a "scaling trick." This involves introducing a scaling parameter a to the activation function, transforming f(z) into a * f(z/a). For sigmoid functions, this effectively increases the derivative magnitude without drastically altering the function's behavior, especially for small inputs. This approach is presented as a simple, parameter-free alternative to more complex methods like batch normalization.

Datasets

The paper uses the CIFAR-10 dataset for its empirical evaluations.

Key Insights and Ideas

  • Two distinct sources of vanishing gradients: The paper's core insight is dissecting the vanishing gradient problem into two separate, addressable causes: individual large parameters and the cumulative effect of many layers. This distinction is crucial because it allows for targeted solutions.
  • Simple remedies are effective: The paper demonstrates that surprisingly simple techniques like L2 regularization and a basic rescaling of activation functions can be highly effective. This challenges the notion that complex architectural changes or entirely new activation functions are always necessary.
  • Sigmoid activations are still relevant: By providing practical solutions to the vanishing gradient problem, the paper makes a strong case for the continued relevance and usability of sigmoid-type activation functions in deep learning, which have been largely sidelined in favor of ReLU.
  • Regularization as a continuous control: The authors highlight that regularization offers a continuous, mathematically justified way to control gradients throughout training, unlike initialization schemes which are applied only once.

Key Results

The paper presents empirical results demonstrating the effectiveness of their proposed remedies:

  • Large-parameters regime: L2 regularization significantly improved accuracy and maintained larger gradients compared to unregularized training, preventing the model from getting stuck.
  • Many-layers regime: The scaling trick (using a=2 for tanh) also led to improved accuracy and larger gradients, effectively mitigating the vanishing gradient problem in deep networks.
  • Combined regime: Both regularization and scaling could be combined, further enhancing performance.
  • Comparison with ReLU: In one experiment, the combined sigmoid approach even outperformed ReLU, suggesting that sigmoid activations are not inherently inferior.
  • Computational Overhead: The paper notes that their remedies do not introduce significant computational overhead, with scaled training being only slightly slower than unscaled training.

Limitations and Future Work

  • No definitive "best" activation: The authors explicitly state that their study is not intended to declare a universally "best" activation function, as the optimal choice depends heavily on the specific task, architecture, and data.
  • Generalizability: While the paper focuses on feedforward networks and tanh activation, the authors believe their concepts can be generalized to recurrent neural networks (RNNs) and other sigmoid-type activations (like softsign).
  • Further research: They suggest exploring mean-field theory approaches for a deeper understanding of vanishing and exploding gradients and investigating other regularization techniques (e.g., L1 regularization). They also pose the open question of which activation function to use in specific settings, emphasizing the need for research beyond anecdotal evidence.

Summary updated

Abstract

Deep learning requires several design choices, such as the nodes' activation functions and the widths, types, and arrangements of the layers. One consideration when making these choices is the vanishing-gradient problem, which is the phenomenon of algorithms getting stuck at suboptimal points due to small gradients. In this paper, we revisit the vanishing-gradient problem in the context of sigmoid-type activation. We use mathematical arguments to highlight two different sources of the phenomenon, namely large individual parameters and effects across layers, and to illustrate two simple remedies, namely regularization and rescaling. We then demonstrate the effectiveness of the two remedies in practice. In view of the vanishing-gradient problem being a main reason why tanh and other sigmoid-type activation has become much less popular than relu-type activation, our results bring sigmoid-type activation back to the table.
BibTeX
@misc{ven2021regularization,
  author = {Leni Ven and Johannes Lederer},
  title = {{Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks}},
  year = {2021},
  eprint = {2106.02260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.02260},
}
Figure