Alias-Free Neural Networks via Anti-Aliased Activations

46 concepts23 papers1 proposal

Modern neural networks are often assumed to be naturally invariant to small transformations. For instance, a classifier should recognize a cat regardless of whether the image is shifted by a single pixel. However, research has shown that standard Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) are surprisingly brittle, often changing their predictions entirely when an image is shifted by a fractional or even an integer amount.

The authors of Why do deep convolutional networks generalize so poorly to small image transformations? identified that this failure stems largely from aliasing—a signal processing artifact where high-frequency information is incorrectly mapped to lower frequencies during sampling. While traditional anti-aliasing focuses on low-pass filtering before downsampling (pooling), a more insidious source of aliasing exists within the network's own layers: the non-linear activation function.

Activations as Frequency Generators

To understand why activations cause aliasing, we must view them through the lens of spectral analysis. A pointwise nonlinearity, such as a ReLU or a sigmoid, acts as a frequency expander. When a signal passes through a nonlinearity, it generates higher-frequency harmonics that were not present in the original input.

As Franzen et al. explain, applying these functions to signals represented by Fourier coefficients generates distortions that can exceed the bandwidth of the underlying representation. If the activation function has sharp transitions or discontinuities—like the "elbow" of a ReLU—it can theoretically introduce infinite frequency components. Michaeli et al. note that these infinite-bandwidth signals inevitably exceed the Nyquist frequency of any discrete sampling grid, causing the signal to "fold back" and alias.

This means that even if a network's input is perfectly filtered, the very first ReLU layer re-introduces high-frequency noise. As the authors of Why do deep convolutional networks generalize so poorly to small image transformations? argue, anti-aliasing cannot be a one-time operation at the input; it must be managed throughout the entire network hierarchy to maintain spatial stability.

The Upsample-Process-Downsample Pipeline

The most robust solution to activation-induced aliasing is to treat the neural network as a continuous signal processor that is only discretized for computation. Karras et al. pioneered this approach in StyleGAN3, redesigning the generative process to be truly alias-free.

The core mechanism involves wrapping every nonlinearity in an upsample-process-downsample sequence:

  1. Upsample: The feature map is increased in resolution to provide "headroom" for the new frequencies the activation will create.
  2. Activate: The nonlinearity is applied in this higher-resolution space.
  3. Downsample: An ideal low-pass filter is applied to remove any frequencies that would alias before the signal is returned to its original resolution.

This ensures that the output remains bandlimited. Raonic et al. applied this principle to create Convolutional Neural Operators (CNOs), demonstrating that enforcing this "continuous-discrete equivalence" allows networks to maintain accuracy across different discretization scales, which is critical for scientific computing and PDE simulations.

Alternative Strategies: Smooth and Polynomial Activations

While upsampling is effective, it is computationally expensive. Researchers have explored alternative activation functions that are inherently more "spectral-friendly."

  • Soft-Capping and Blurring: Hossain et al. proposed AA-ReLU, which incorporates a smooth sinusoidal roll-off. By replacing hard corners with soft caps, the activation acts as its own secondary low-pass filter, suppressing the re-introduction of high-frequency noise.
  • Polynomial Activations: Michaeli et al. suggested using low-degree polynomials. Because a polynomial of degree $d$ expands the frequency spectrum by exactly a factor of $d$, the required upsampling rate becomes finite and mathematically predictable. This allows for precise control over bandwidth expansion without the guesswork of infinite-bandwidth functions like ReLU.
  • Differentiable Filters: In audio synthesis, Shan et al. demonstrated that incorporating differentiable anti-aliasing filters during training ensures that generated waveforms remain physically plausible and free of spectral folding artifacts.

Impact Across Domains

The consequences of ignoring activation-induced aliasing vary by field, but the solution remains consistent:

  • Computer Vision: In Alias-Free Latent Diffusion Models, Zhou et al. show that anti-aliasing eliminates "texture sticking" in videos, where fine details appear glued to the screen rather than moving with objects.
  • Audio Generation: Lee et al. found that integrating low-pass filters within the upsampling modules of BigVGAN resulted in significantly crisper high-frequency output by preventing digital distortion.
  • Scientific Computing: Bartolucci et al. argue that alias-free activations are essential for Neural Operators to achieve true resolution invariance, preventing high-frequency artifacts from corrupting physical simulations like fluid dynamics.

By acknowledging that nonlinearities are noise generators, researchers are moving toward architectures that respect the fundamental limits of the sampling theorem, leading to models that are more robust, stable, and physically accurate.

Key Foundational Work

Impact of Aliasing on Deep CNN-Based End-to-End Acoustic Models (2018) — contributed 4 concept elements to this direction.
Why do deep convolutional networks generalize so poorly to small image transformations? (2018) — contributed 3 concept elements to this direction.
Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function (2021) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_activation_induced_aliasing_f24b1d96,
  title = {activation-induced aliasing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-induced-aliasing/rdr_f24b1d96e7e443f8be24bab1f1b5e985},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_activation_induced_aliasing_f24b1d96_v1,
  title = {activation-induced aliasing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-induced-aliasing/rdr_f24b1d96e7e443f8be24bab1f1b5e985/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake