Modern neural networks are often assumed to be naturally invariant to small transformations. For instance, a classifier should recognize a cat regardless of whether the image is shifted by a single pixel. However, research has shown that standard Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) are surprisingly brittle, often changing their predictions entirely when an image is shifted by a fractional or even an integer amount.
The authors of Why do deep convolutional networks generalize so poorly to small image transformations? identified that this failure stems largely from aliasing—a signal processing artifact where high-frequency information is incorrectly mapped to lower frequencies during sampling. While traditional anti-aliasing focuses on low-pass filtering before downsampling (pooling), a more insidious source of aliasing exists within the network's own layers: the non-linear activation function.
Activations as Frequency Generators
To understand why activations cause aliasing, we must view them through the lens of spectral analysis. A pointwise nonlinearity, such as a ReLU or a sigmoid, acts as a frequency expander. When a signal passes through a nonlinearity, it generates higher-frequency harmonics that were not present in the original input.
As Franzen et al. explain, applying these functions to signals represented by Fourier coefficients generates distortions that can exceed the bandwidth of the underlying representation. If the activation function has sharp transitions or discontinuities—like the "elbow" of a ReLU—it can theoretically introduce infinite frequency components. Michaeli et al. note that these infinite-bandwidth signals inevitably exceed the Nyquist frequency of any discrete sampling grid, causing the signal to "fold back" and alias.
This means that even if a network's input is perfectly filtered, the very first ReLU layer re-introduces high-frequency noise. As the authors of Why do deep convolutional networks generalize so poorly to small image transformations? argue, anti-aliasing cannot be a one-time operation at the input; it must be managed throughout the entire network hierarchy to maintain spatial stability.
The Upsample-Process-Downsample Pipeline
The most robust solution to activation-induced aliasing is to treat the neural network as a continuous signal processor that is only discretized for computation. Karras et al. pioneered this approach in StyleGAN3, redesigning the generative process to be truly alias-free.
The core mechanism involves wrapping every nonlinearity in an upsample-process-downsample sequence:
- Upsample: The feature map is increased in resolution to provide "headroom" for the new frequencies the activation will create.
- Activate: The nonlinearity is applied in this higher-resolution space.
- Downsample: An ideal low-pass filter is applied to remove any frequencies that would alias before the signal is returned to its original resolution.
This ensures that the output remains bandlimited. Raonic et al. applied this principle to create Convolutional Neural Operators (CNOs), demonstrating that enforcing this "continuous-discrete equivalence" allows networks to maintain accuracy across different discretization scales, which is critical for scientific computing and PDE simulations.
Alternative Strategies: Smooth and Polynomial Activations
While upsampling is effective, it is computationally expensive. Researchers have explored alternative activation functions that are inherently more "spectral-friendly."
- Soft-Capping and Blurring: Hossain et al. proposed AA-ReLU, which incorporates a smooth sinusoidal roll-off. By replacing hard corners with soft caps, the activation acts as its own secondary low-pass filter, suppressing the re-introduction of high-frequency noise.
- Polynomial Activations: Michaeli et al. suggested using low-degree polynomials. Because a polynomial of degree $d$ expands the frequency spectrum by exactly a factor of $d$, the required upsampling rate becomes finite and mathematically predictable. This allows for precise control over bandwidth expansion without the guesswork of infinite-bandwidth functions like ReLU.
- Differentiable Filters: In audio synthesis, Shan et al. demonstrated that incorporating differentiable anti-aliasing filters during training ensures that generated waveforms remain physically plausible and free of spectral folding artifacts.
Impact Across Domains
The consequences of ignoring activation-induced aliasing vary by field, but the solution remains consistent:
- Computer Vision: In Alias-Free Latent Diffusion Models, Zhou et al. show that anti-aliasing eliminates "texture sticking" in videos, where fine details appear glued to the screen rather than moving with objects.
- Audio Generation: Lee et al. found that integrating low-pass filters within the upsampling modules of BigVGAN resulted in significantly crisper high-frequency output by preventing digital distortion.
- Scientific Computing: Bartolucci et al. argue that alias-free activations are essential for Neural Operators to achieve true resolution invariance, preventing high-frequency artifacts from corrupting physical simulations like fluid dynamics.
By acknowledging that nonlinearities are noise generators, researchers are moving toward architectures that respect the fundamental limits of the sampling theorem, leading to models that are more robust, stable, and physically accurate.
Key Foundational Work
Impact of Aliasing on Deep CNN-Based End-to-End Acoustic Models (2018) — contributed 4 concept elements to this direction.
Why do deep convolutional networks generalize so poorly to small image transformations? (2018) — contributed 3 concept elements to this direction.
Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function (2021) — contributed 3 concept elements to this direction.