Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks

Christian H.x. Ali Mehmeti-Göpel, David Hartmann, Michael WandICLR 2021 · OpenReview

Abstract: This paper introduces harmonic distortion analysis to explain why deep neural networks become increasingly difficult to train as they get deeper. The authors show that nonlinearities act like audio clipping, creating high-frequency "harmonics" (a phenomenon they call blueshift) that make the loss landscape jagged. Crucially, they demonstrate that residual connections act as a filter to dampen these frequencies and restore trainability.

Introduction

We know that deep networks are highly expressive, but they are notoriously hard to train. Standard explanations usually focus on "local" issues like vanishing or exploding gradients (the Jacobian of the layers). However, these local metrics do not fully capture the "global" geometry of the landscape—why does the loss surface look like a smooth bowl for shallow networks but a jagged, chaotic mountain range for deep ones?

3D landscape showing how ReLU introduces roughness while ResNet connections restore smoothness.

The authors seek to provide an analytical, spectral explanation for this "roughness." They aim to understand how architectural choices—like the type of activation function or the use of skip connections—directly change the frequency content of the mapping from weights to outputs.

Methods: Harmonic Distortion and Blueshift

The researchers treat a neural network like a signal processing system. They analyze the weights-to-outputs (w-o) function by taking 1D slices through the parameter space and performing Fourier analysis.

Key technical components include:

  • Harmonic Distortion Analysis: By approximating nonlinearities (like ReLU or Tanh) as polynomials, they show that passing a "signal" (the pre-activations) through a nonlinearity creates higher-frequency components.
  • The "Blueshift" Model: Just as a distorted guitar amplifier adds high-pitched overtones to a note, each layer in a network shifts the power spectrum of the function toward higher frequencies.
  • Fractal Analysis: They quantify "roughness" using a fractal coefficient ($h$). A higher $h$ means a smoother function (lower frequencies dominate), while a lower $h$ indicates a "rougher," more chaotic function.
  • ResNet Analysis: They identify two specific mechanisms in Residual Networks that counter blueshift:
  • Exponential Downweighting (ED): Skip connections reduce the relative influence of the "noisy" nonlinear branches.
  • Frequency-Dependent Signal-Averaging (FDSA): Summing the identity and the residual branch naturally dampens high frequencies because those harmonics are less correlated between the two paths.

Theoretical Insights

  • Nonlinearities are "Clippers": Applying a ReLU is mathematically similar to clipping an audio signal. This creates "ringing" artifacts in the spectral domain, adding high-frequency "noise" to the parameter landscape.
  • The Depth Penalty: In a standard feedforward network, this blueshift compounds. Each layer takes the already-shifted spectrum of the previous layer and shifts it even further. This is why deep "plain" networks eventually become untrainable—the landscape becomes too jagged for gradient descent to navigate.
  • Exploding Gradients as High-Pass Filters: The authors provide a fresh perspective on exploding gradients. Since taking a derivative is a high-pass filter (it multiplies Fourier coefficients by frequency $k$), a "blueshifted" landscape naturally leads to massive gradients.
  • ResNets as Low-Pass Filters: Residual connections aren't just about "letting gradients flow." Spectrally, they act as a mechanism to keep the power spectrum concentrated in low frequencies, effectively smoothing the optimization landscape.

Key Results

Power spectrum across layers for different activations and architectures. Plain networks (top) show severe blueshift with ReLU, while residual connections (bottom) dampen high frequencies.
  • Activation Comparison: Softplus creates the least blueshift (smoothest), followed by Tanh, with ReLU creating the most high-frequency distortion. This matches the empirical observation that smoother activations can sometimes be easier to train in very deep settings.
  • ResNet Smoothness: In experiments with a 50-layer Toy-CNN, the fractal coefficient ($h$) for plain networks dropped sharply with depth (increasing roughness). In contrast, ResNets maintained a much higher, more stable $h$, confirming the smoothing effect of skip connections.
  • Trainability Correlation: The authors showed a clear correlation between the amount of blueshift at initialization and the final training accuracy on datasets like CIFAR-10. Networks with excessive high-frequency content (low $h$) failed to converge, while those with dampened harmonics (via skip connections or smoother activations) trained successfully.

Limitations and Future Work

  • Polynomial Approximation: The core theory relies on polynomial approximations of nonlinearities. While the authors show this holds up empirically for ReLU, a more rigorous treatment of non-polynomial, non-differentiable functions is still needed.
  • Initialization Focus: Most of the spectral analysis is performed at or near initialization. While they provide some evidence that these effects persist during training, the "spectral evolution" of a network as it learns is a complex topic for future study.
  • Architectural Scope: The paper focuses on feedforward and residual structures. It does not explicitly model other common components like max-pooling or attention mechanisms, which likely have their own unique spectral signatures.

Summary updated

Abstract

In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. Each nonlinear layer creates higher-frequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape. Unlike differential models (such as vanishing gradients, sharpness), this provides a more global view of how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.
BibTeX
@inproceedings{mehmetigopel2021ringing,
  author = {Christian H.X. Ali Mehmeti-Göpel and David Hartmann and Michael Wand},
  title = {{Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=TaYhv-q1Xit},
}
Figure