TL;DR
This paper surveys common activation functions (AFs) in neural networks, highlighting their importance in introducing non-linearity and enabling complex pattern recognition. It provides a practical guide for researchers on selecting appropriate AFs based on application, network architecture, and potential issues like vanishing gradients.
What Problem Does This Paper Address?
Deep learning architectures, built upon Artificial Neural Networks (ANNs), have seen tremendous growth due to their ability to solve complex real-world problems. A crucial component of these ANNs is the activation function (AF), which introduces non-linearity, allowing the network to learn intricate relationships between input features and outputs. However, the sheer variety of AFs and the factors influencing their selection (application type, network design, data complexity, optimizers) can be overwhelming for researchers. This paper aims to address this by providing a survey of widely used AFs and offering guidance on choosing the most suitable one for a given problem domain.
Methods
The paper presents a survey-style review of various activation functions. It categorizes them and discusses their properties, advantages, and disadvantages. The authors draw upon existing literature to analyze the performance of different AFs across various applications and network architectures. The core methodology involves synthesizing findings from previous research to offer practical recommendations and guidelines for selecting AFs. They discuss key considerations such as differentiability, the vanishing/exploding gradient problem, and neuron sparsity.
Datasets
The paper references several datasets used in the literature it surveys, including:
- microRNA features for cancer classification.
- TIMIT and WSJ datasets for speech recognition.
- MNIST dataset for handwritten digit classification.
- CIFAR-10 and CIFAR-100 datasets for image classification.
- IMDB and Movie Review datasets for sentiment analysis.
The paper does not introduce new datasets but rather synthesizes results from studies that utilized these existing benchmarks.
Key Insights and Ideas
- Activation functions are the backbone of non-linearity: Without AFs, ANNs would essentially perform linear transformations, severely limiting their ability to capture complex patterns.
- No one-size-fits-all AF: The choice of AF is highly context-dependent, influenced by the specific problem, network architecture, and even the optimizer used.
- Vanishing gradients are a major challenge: Sigmoid and Tanh functions, while historically important, are prone to vanishing gradients, hindering learning in deep networks. ReLU and its variants offer a solution by having a non-zero gradient for positive inputs.
- ReLU's dominance and its "dead neuron" issue: ReLU is a popular default choice for hidden layers due to its simplicity and effectiveness. However, its zero gradient for negative inputs can lead to "dead neurons," which Leaky ReLU and other variants aim to address.
- Architectural considerations matter: The choice of AF for output layers differs from hidden layers. Softmax is common for multi-class classification, while sigmoid is used for binary classification. For hidden layers, non-linearity, unboundedness, and differentiability are key.
Key Results
The paper synthesizes results from various studies, highlighting the following:
- Accuracy: In a study on CIFAR-10, ReLU and Leaky ReLU achieved over 70% accuracy, outperforming Sigmoid and Tanh. Another study showed ReLU achieving 98.536% accuracy with the Adam optimizer for cancer classification.
- Convergence Speed: ReLU has been shown to converge significantly faster than Tanh (e.g., six times faster on CIFAR-10). Exponential Linear Unit (ELU) also speeds up convergence due to its negative values acting like batch normalization.
- Performance on Specific Tasks:
- Sigmoid is often used in Logistic Regression and for binary classification output layers.
- Tanh is predominantly used in LSTM cells for NLP tasks.
- ReLU and its variants are highly effective for image classification tasks (CNNs).
- For speech recognition, ELU showed better results on larger datasets compared to ReLU.
- For object, face, and text detection, ReLU is generally best suited, while SELU and Maxout are better for sound/speech detection.
Limitations and Future Work
- Limited Scope: The paper acknowledges that due to space constraints, not all AFs could be discussed in detail.
- Future Work: The authors suggest a future work that would involve a comprehensive analysis of all AFs, including their advantages and disadvantages in specific situations. They also propose a more detailed, implementation-level comparison of various AFs to provide enhanced guidelines.
Summary updated