Nonlinear Activation Free Networks for Image Restoration

This direction is split into other directions: activation-free gating. Its last version stays readable and citable here.

Version 1 of 122 papers36 conceptsfrozen 2026-10-04

For decades, the "nonlinear activation function" (such as ReLU, GeLU, or Sigmoid) has been considered the indispensable engine of neural networks. The logic was simple: without these functions, a multi-layer network collapses into a single linear transformation, incapable of learning complex patterns. However, recent breakthroughs in image restoration have challenged this dogma, demonstrating that traditional activations can be entirely removed and replaced by simpler multiplicative operations without losing representational power.

This architectural shift, pioneered by Liangyu Chen 0002, Xiaojie Chu, and Xiangyu Zhang in their work on Simple Baselines for Image Restoration, has led to the NAFNet (Nonlinear Activation Free Network) family. These models achieve state-of-the-art results in denoising and deblurring while being significantly more efficient than their predecessors.

The Intuition: Nonlinearity via Multiplication

The core insight behind NAFNet is that nonlinearity doesn't have to come from a dedicated "activation layer." Instead, it can be a byproduct of how features interact.

In a standard Gated Linear Unit (GLU), a feature map is split into two; one half is passed through a Sigmoid function (the "gate") and then multiplied by the other half. Liangyu Chen 0002 and colleagues observed that the Sigmoid itself—a computationally expensive nonlinear function—might be redundant for low-level vision tasks.

By simply splitting the feature map and performing element-wise multiplication between the two halves (a mechanism they call a Simple Gate), the network introduces the necessary nonlinearity through basic arithmetic. This approach maintains the network's ability to model complex signal distortions while stripping away the overhead of traditional activations.

Efficiency Through Simplification

The primary advantage of removing activations is a dramatic reduction in architectural complexity. Traditional models often stack layers of convolutions, normalization, and activations. NAFNet simplifies this into a streamlined block that prioritizes multiplicative interactions.

  1. Lower Parameter Count: NAFSSR, an adaptation for stereo image super-resolution, achieved top performance in the NTIRE 2022 challenge while using up to 79% fewer parameters than competing models.
  2. Hardware Friendliness: Because the architecture relies on standard convolutions and element-wise multiplication, it is highly optimized for modern hardware. Minghao Han and Shuhang Gu demonstrated that these blocks allow convolutional models to match the quality of Transformers in image compression while reducing inference latency by over 20% (Causal Context Adjustment Loss for Learned Image Compression).
  3. Speed: In specialized domains like spacecraft imagery, the SAFFN model by Jingfan Yang and Hu Gao utilized activation-free designs to run 30 times faster than general-purpose denoising models.

Generalizing Across Domains

The "activation-free" philosophy has proven remarkably versatile, extending beyond standard RGB image restoration:

The Security Trade-off

While activation-free networks offer efficiency and performance, they are not without drawbacks. Research by Shashank Agnihotri and Margret Keuper has revealed an "unreasonable vulnerability" to adversarial attacks. By removing the "squashing" effects of functions like ReLU, these networks can become overly sensitive to small input perturbations, creating potential security holes in dense prediction tasks (On the unreasonable vulnerability of transformers for image restoration).

Despite this, the trend toward activation-free architectures represents a fundamental rethinking of neural network design: for many tasks, the "simple" interaction of features is more powerful—and far more efficient—than the complex nonlinearities we once thought were mandatory.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_nonlinear_activation_free_networks_for_i_aa4de72a,
  title = {Nonlinear Activation Free Networks for Image Restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/nonlinear-activation-free-networks-for-image-restoration/rdr_aa4de72a79084bf3bc39ae6ab5fa9e8b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_nonlinear_activation_free_networks_for_i_aa4de72a_v1,
  title = {Nonlinear Activation Free Networks for Image Restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/nonlinear-activation-free-networks-for-image-restoration/rdr_aa4de72a79084bf3bc39ae6ab5fa9e8b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}