For decades, the "nonlinear activation function" (such as ReLU, GeLU, or Sigmoid) has been considered the indispensable engine of neural networks. The logic was simple: without these functions, a multi-layer network collapses into a single linear transformation, incapable of learning complex patterns. However, recent breakthroughs in image restoration have challenged this dogma, demonstrating that traditional activations can be entirely removed and replaced by simpler multiplicative operations without losing representational power.
This architectural shift, pioneered by Liangyu Chen 0002, Xiaojie Chu, and Xiangyu Zhang in their work on Simple Baselines for Image Restoration, has led to the NAFNet (Nonlinear Activation Free Network) family. These models achieve state-of-the-art results in denoising and deblurring while being significantly more efficient than their predecessors.
The Intuition: Nonlinearity via Multiplication
The core insight behind NAFNet is that nonlinearity doesn't have to come from a dedicated "activation layer." Instead, it can be a byproduct of how features interact.
In a standard Gated Linear Unit (GLU), a feature map is split into two; one half is passed through a Sigmoid function (the "gate") and then multiplied by the other half. Liangyu Chen 0002 and colleagues observed that the Sigmoid itself—a computationally expensive nonlinear function—might be redundant for low-level vision tasks.
By simply splitting the feature map and performing element-wise multiplication between the two halves (a mechanism they call a Simple Gate), the network introduces the necessary nonlinearity through basic arithmetic. This approach maintains the network's ability to model complex signal distortions while stripping away the overhead of traditional activations.
Efficiency Through Simplification
The primary advantage of removing activations is a dramatic reduction in architectural complexity. Traditional models often stack layers of convolutions, normalization, and activations. NAFNet simplifies this into a streamlined block that prioritizes multiplicative interactions.
- Lower Parameter Count: NAFSSR, an adaptation for stereo image super-resolution, achieved top performance in the NTIRE 2022 challenge while using up to 79% fewer parameters than competing models.
- Hardware Friendliness: Because the architecture relies on standard convolutions and element-wise multiplication, it is highly optimized for modern hardware. Minghao Han and Shuhang Gu demonstrated that these blocks allow convolutional models to match the quality of Transformers in image compression while reducing inference latency by over 20% (Causal Context Adjustment Loss for Learned Image Compression).
- Speed: In specialized domains like spacecraft imagery, the SAFFN model by Jingfan Yang and Hu Gao utilized activation-free designs to run 30 times faster than general-purpose denoising models.
Generalizing Across Domains
The "activation-free" philosophy has proven remarkably versatile, extending beyond standard RGB image restoration:
- Signal Processing: In medical contexts, Chanho Lim and Zhengming Ding developed Cardio-NAFNet to denoise ECG recordings. They found that multiplicative gating preserves fine-grained temporal features better than standard convolutions, leading to an 11dB improvement in signal-to-noise ratio (Generalizability and Clinical Implications of Electrocardiogram Denoising with Cardio-NAFNet).
- Generative Models: Danilo Comminiello applied these principles to Diffusion Probabilistic Models for document enhancement, reducing the "hallucination" of characters by simplifying the restoration backbone (NAF-DPM).
- Stereo Vision: The NTIRE challenges in 2022 and 2023 (Longguang Wang et al.) highlighted a shift where activation-free networks overtook traditional CNNs, proving that the quality of feature extraction and cross-view fusion is more critical than the complexity of nonlinear layers (NTIRE 2023 Challenge on Stereo Image Super-Resolution).
The Security Trade-off
While activation-free networks offer efficiency and performance, they are not without drawbacks. Research by Shashank Agnihotri and Margret Keuper has revealed an "unreasonable vulnerability" to adversarial attacks. By removing the "squashing" effects of functions like ReLU, these networks can become overly sensitive to small input perturbations, creating potential security holes in dense prediction tasks (On the unreasonable vulnerability of transformers for image restoration).
Despite this, the trend toward activation-free architectures represents a fundamental rethinking of neural network design: for many tasks, the "simple" interaction of features is more powerful—and far more efficient—than the complex nonlinearities we once thought were mandatory.