TL;DR
This paper introduces Fractional Adaptive Linear Units (FALUs), a novel class of activation functions that generalize existing ones like Sigmoid, ReLU, GELU, and Swish. By leveraging fractional calculus and introducing a small number of trainable parameters, FALUs offer a flexible and automated way to optimize activation functions, leading to improved performance across various neural network architectures and tasks.
What Problem Does This Paper Address?
The paper addresses the relative stagnation in activation function (AF) research despite their crucial role in neural network performance. While many architectural and regularization techniques have been developed, AFs have largely remained static, with ReLU being the de facto standard. Existing attempts to create more adaptive AFs often introduce complexity or limitations. The authors aim to bridge this gap by proposing a unified, flexible framework that can encompass and extend a wide range of existing AFs, offering a more automated and performant solution.
Methods
The core innovation is the introduction of Fractional Adaptive Linear Units (FALUs). These are defined by extending the concept of derivatives to fractional orders, inspired by fractional calculus. Specifically, FALUs are formulated as a dynamic generalization of the Swish activation function, incorporating two tunable parameters: $\alpha$, a real-valued fractional derivative order, and $\beta$, a scaling parameter.
The paper derives the mathematical formulation of FALUs using fractional calculus, including approximations for efficient implementation in backpropagation. They also explore how different values of $\alpha$ and $\beta$ can generate various known activation functions (e.g., $\alpha=0$ and $\beta=1$ for Swish, $\alpha=0$ and $\beta \approx 1.702$ for GELU) and a rich family of novel, interpolated AFs. The authors provide explicit update rules and computationally tractable approximations for these parameters.
Datasets
The paper evaluates FALUs on the following datasets:
- MNIST: A standard dataset for handwritten digit classification.
- Fashion-MNIST: A dataset of clothing images, serving as a more challenging alternative to MNIST.
- CIFAR-10: A dataset of small color images in 10 classes.
- ImageNet: A large-scale dataset for image classification.
Key Insights and Ideas
- Unified Activation Function Framework: FALUs provide a single mathematical framework that can represent a wide spectrum of existing activation functions, from simple linear units to more complex smooth functions like GELU and Swish, and even interpolations between them.
- Fractional Calculus as a Tool for AF Design: The paper demonstrates the utility of fractional calculus in creating novel and flexible activation functions, opening up a new avenue for AF research.
- Automated AF Optimization: By introducing trainable parameters ($\alpha$ and $\beta$), FALUs allow for automated tuning of the activation function's shape and behavior to suit specific datasets and tasks, reducing the need for manual AF selection.
- Performance Gains with Minimal Overhead: Despite their increased flexibility, FALUs introduce only a small number of additional trainable parameters, making them computationally efficient and easy to integrate into existing deep learning workflows.
- Diverse AF Convergence: Experiments show that when trained, FALUs can converge to various forms, including ReLU, sigmoid, and Gaussian shapes, indicating their ability to adapt to different network needs.
Key Results
- Improved Accuracy: FALUs consistently matched or outperformed baseline activation functions (including state-of-the-art ones like GELU) across all tested datasets and model architectures.
- On MNIST, FALU improved accuracy by 0.1% and 0.2% on compact CNN models compared to ReLU.
- On CIFAR-10, FALU yielded a 1.39% error reduction on a compact ResNet18 model compared to ReLU.
- On Fashion-MNIST, FALU improved accuracy by roughly 1% over ReLU across different CNN architectures.
- On ImageNet, FALU showed performance gains over baseline AFs, with improvements being more pronounced for larger model sizes.
- Flexibility: The parameter $\alpha$ allows for the generation of a rich family of activation functions, visualized in figures, demonstrating smooth transitions between different functional forms.
- Minimal Parameter Overhead: The additional parameters for FALU are negligible compared to the overall model parameters, ensuring computational efficiency.
Limitations and Future Work
- Approximation for Backpropagation: While the paper derives exact formulas for fractional derivatives, it relies on approximations for practical implementation in backpropagation. The paper acknowledges that these approximations might have limitations in certain scenarios.
- Parameter Initialization and Range: The paper mentions initializing $\alpha$ in $[0, 1]$ and $\beta$ in $[1, 10]$ and clamping them during training. Further research could explore optimal initialization strategies and broader parameter ranges.
- Exploration of Broader Fractional Orders: The paper primarily focuses on $\alpha$ in the range $[0, 2]$. Future work could investigate the impact of higher fractional orders.
- Further Architectural Exploration: While tested on various architectures, exploring FALUs in more complex and specialized network designs could reveal further benefits.
Summary updated