TL;DR
This paper provides a rigorous mathematical foundation for the Frequency Principle (F-Principle), the observation that deep neural networks (DNNs) learn low-frequency components of a target function before high-frequency ones. By analyzing the training dynamics across initial, intermediate, and final stages, the authors prove that this "low-to-high" priority is a universal feature of general DNNs, regardless of the specific activation function or loss type used.
What Problem Does This Paper Address?
While the Universal Approximation Theorem tells us that a sufficiently wide neural network can represent almost any function, it doesn't explain how the network finds that representation during training. Empirical studies have long noted a curious bias: networks seem to fit the "smooth" parts of a function (low frequencies) very quickly, while the "noisy" or sharp details (high frequencies) take much longer to emerge.
This phenomenon is known as the Frequency Principle. While previous research had explored this for simple two-layer networks or specific activation functions like ReLU, a unified theory for general, multi-layer DNNs was missing. The authors aim to bridge this gap, providing a rigorous proof that explains why this spectral bias is a fundamental property of gradient-based training in deep learning.
Methods
The authors employ Fourier Analysis to dissect the training dynamics of DNNs. The core of their approach involves:
- Frequency Decomposition: They split the Mean Squared Error (MSE) loss into two parts: $L^-_\eta$ (contributions from low frequencies within a radius $\eta$) and $L^+_\eta$ (contributions from high frequencies outside that radius).
- Regularity Analysis: Using the "Japanese bracket" $\langle \xi \rangle = (1 + |\xi|^2)^{1/2}$, they characterize the smoothness (regularity) of the network's output. They show that the regularity of the DNN architecture directly translates to the decay rate of the loss function in the frequency domain.
- Gradient Flow Modeling: They treat the training process as a continuous gradient flow. By estimating the ratio of the change in high-frequency loss versus total loss, they prove that the high-frequency change is significantly smaller, especially early in training.
- Generalization: Unlike previous works, this theory applies to networks with $(H-1)$ hidden layers, various activation functions (ReLU, sigmoid, tanh), and a broad class of $L^p$ loss functions.
Datasets
As a theoretical paper, the authors primarily use synthetic data to illustrate their theorems:
- Target Function: A complex 1D function composed of 500 sine waves: $f(x) = \sum_{j=1}^{500} \sin(jx/10)/j$.
- Training Setup: They trained a tanh-activated DNN (widths 1-200-50-1) using the Adam optimizer on 300 uniformly sampled points.
- Loss Functions: They tested both standard MSE ($L^2$) and $L^4$ loss to demonstrate the theory's versatility.
Key Insights and Ideas
- The "Half-Life" Metric: In the intermediate stage of training, the loss doesn't always decrease monotonically for every frequency. To handle this, the authors introduced a "half-life" period—the time it takes for the total error to be reduced by half. They proved that over this window, the reduction in error is dominated by low frequencies.
- Regularity is the Key: The fundamental reason for the F-Principle is that neural networks are inherently "regular" (smooth) functions. Because the network's gradient with respect to its parameters is smoother than the target function's high-frequency components, the optimizer naturally prioritizes the components it can "see" most clearly: the low frequencies.
- Universality: The F-Principle isn't an accident of using ReLU or MSE. It is "universal" in the sense that it persists across different architectures and loss landscapes, provided the network maintains some level of regularity.
Key Results
The paper presents three main theorems corresponding to the stages of training:
- Initial Stage: The ratio of high-frequency loss change to total loss change is bounded by $C\eta^{-m}$, where $m$ relates to the network's regularity. As the frequency $\eta$ increases, this ratio drops to near zero, meaning high frequencies barely change at the start.
- Intermediate Stage: Even when the dynamics become complex, the authors prove that the difference in MSE over a significant training period is dominated by low frequencies.
- Final Stage: As the network approaches a non-degenerate global minimizer, the F-Principle still holds, ensuring that the very last bits of error being squeezed out are the highest frequencies.
Numerical results (Figures 1 and 2 in the paper) confirm these theorems, showing that the relative error for the first peak frequency of the target function drops to near zero long before the higher-frequency peaks are even addressed.
Limitations and Future Work
- Bounded Trajectories: The proofs rely on Assumption 3, which assumes the parameters $\theta(t)$ stay within a bounded region. While common in practice, proving this for all possible initializations remains a challenge.
- Discrete vs. Continuous: The theory is developed in the continuous frequency domain. While the numerical examples use discrete Fourier transforms, a more explicit bridge between discrete data samples and continuous frequency theory could be explored.
- Optimization Algorithms: The theory focuses on gradient flow. While numerical tests used Adam, the theoretical proofs are most direct for standard Gradient Descent. Exploring how different optimizers (like those with momentum or adaptive rates) specifically accelerate or decelerate this frequency-based learning could be a fruitful next step.
Summary updated