Serf: Towards better training of deep neural networks using log-Softplus ERror activation Function

Sayan Nag, Mayukh Bhattacharyya, Anuraag Mukherjee, Rohit KunduWACV 2023 · OpenReview

Abstract

Activation functions play a pivotal role in determining the training dynamics and neural network performance. The widely adopted activation function ReLU despite being simple and effective has few disadvantages including the Dying ReLU problem. In order to tackle such problems, we propose a novel activation function called Serf which is self-regularized and non-monotonic in nature. Like Mish, Serf also belongs to the Swish family of functions. Based on several experiments on computer vision (image classification and object detection) and natural language processing (machine translation, sentiment classification and multi-modal entailment) tasks with different state-of-the-art architectures, it is observed that Serf vastly outperforms ReLU (baseline) and other activation functions including both Swish and Mish, with a markedly bigger margin on deeper architectures. Ablation studies further demonstrate that Serf based architectures perform better than those of Swish and Mish in varying scenarios, validating the effectiveness and compatibility of Serf with varying depth, complexity, optimizers, learning rates, batch sizes, initializers and dropout rates. Finally, we investigate the mathematical relation between Swish and Serf, thereby showing the impact of pre-conditioner function ingrained in the first derivative of Serf which provides a regularization effect making gradients smoother and optimization faster.

Introduction

For years, the Rectified Linear Unit (ReLU) has been the default choice for deep learning. However, it suffers from two major flaws: the "Dying ReLU" problem, where neurons become inactive due to zero gradients for negative inputs, and a lack of smoothness at zero, which can cause inconsistencies during optimization. While smooth, non-monotonic functions like Swish and Mish were designed to address these issues, there remains potential to improve how these functions "pre-condition" gradients for the complex loss landscapes of very deep networks. This paper introduces Serf, a novel activation function designed to overcome these limitations.

Methods

The proposed Serf (log-Softplus ERror) activation function is defined as: $$f(x) = x \cdot \text{erf}(\ln(1 + e^x))$$

Serf exhibits several key mathematical properties that aid training:

  • Non-monotonicity: It features a small negative "bump" (minimum $\approx -0.3484$), preserving small negative weights and improving expressivity.
  • Smoothness: As a $C^\infty$ smooth function, it avoids the optimization jitters associated with ReLU's non-differentiability.
  • Self-Gating: It utilizes the input $x$ to gate itself via a non-linear transformation.
  • Pre-conditioning: The first derivative of Serf contains a "pre-conditioner" term that smooths gradients, effectively acting as a locally adaptive learning rate.
Comparison of Serf, Mish, and Swish functions and their derivatives.

Experimental Results

The effectiveness of Serf was validated across a broad spectrum of benchmarks, including ImageNet, CIFAR-10/100, MS-COCO for object detection, and various NLP tasks like sentiment analysis and machine translation.

Performance Highlights

  • ImageNet: Using ResNet-50, Serf achieved a 75.34% Top-1 accuracy, significantly higher than ReLU's 74.16%.
  • CIFAR-100: Serf consistently outperformed Mish and ReLU across multiple architectures (ResNet, WideResNet, DenseNet, Inception).
  • Object Detection: On MS-COCO, Serf improved the mAP of YOLOv3 to 33.4 from the LeakyReLU baseline of 32.5.
  • Depth Scalability: A critical finding is that Serf’s performance advantage over ReLU and Mish increases with network depth, making it ideal for modern, massive architectures.

Landscape Smoothness and Robustness

Visualizations of the output landscapes demonstrate that Serf produces a much smoother surface compared to the jagged transitions of ReLU, facilitating easier discovery of global minima by optimizers.

Visualization of output landscapes showing the smoothing effect of Serf.

Ablation studies on MNIST further confirmed that Serf is less sensitive to variations in hyperparameters such as learning rates, dropout rates, and initialization schemes compared to its predecessors.

Conclusion

Serf represents a significant step forward in activation function design, leveraging the error function and log-softplus to provide superior regularization and gradient pre-conditioning. While it carries a slightly higher computational cost than ReLU, the performance gains in deep architectures and complex tasks make it a compelling choice for modern deep learning. Future work includes exploring parameterized versions of Serf and its application in generative tasks.

Summary updated

Abstract

Activation functions play a pivotal role in determining the training dynamics and neural network performance. The widely adopted activation function ReLU despite being simple and effective has few disadvantages including the Dying ReLU problem. In order to tackle such problems, we propose a novel activation function called Serf which is self-regularized and non-monotonic in nature. Like Mish, Serf also belongs to the Swish family of functions. Based on several experiments on computer vision (image classification and object detection) and natural language processing (machine translation, sentiment classification and multi-modal entailment) tasks with different state-of-the-art architectures, it is observed that Serf vastly outperforms ReLU (baseline) and other activation functions including both Swish and Mish, with a markedly bigger margin on deeper architectures. Ablation studies further demonstrate that Serf based architectures perform better than those of Swish and Mish in varying scenarios, validating the effectiveness and compatibility of Serf with varying depth, complexity, optimizers, learning rates, batch sizes, initializers and dropout rates. Finally, we investigate the mathematical relation between Swish and Serf, thereby showing the impact of pre-conditioner function ingrained in the first derivative of Serf which provides a regularization effect making gradients smoother and optimization faster.
BibTeX
@inproceedings{nag2023serf,
  author = {Sayan Nag and Mayukh Bhattacharyya and Anuraag Mukherjee and Rohit Kundu},
  title = {{Serf: Towards better training of deep neural networks using log-Softplus ERror activation Function}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=m9fnarpEcu},
}
Figure