Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data

Guillaume Braun, Bruno Loureiro, Ha Quang Minh, Masaaki Imaizumi

Introduction

In the quest to understand the "scaling laws" of deep learning—the empirical relationships predicting model performance based on data, compute, and time—theoretical models often fall short by assuming uniform, or "isotropic," data. Real-world data is inherently anisotropic, where a few principal components dominate the variance while a long "tail" of directions contains minimal information. This paper addresses this gap by investigating how the spectral decay of input data reshapes the fundamental dynamics of nonlinear learning.

Problem Formulation: Phase Retrieval with Anisotropic Data

The authors focus on the Phase Retrieval problem, a canonical nonlinear task where the objective is to recover a signal $w^*$ from quadratic measurements $y = \langle x, w^* \rangle^2$. To mirror realistic data distributions, the input $x$ is assumed to have a covariance matrix $Q$ with eigenvalues following a power law: $\lambda_i \propto i^{-a}$. The exponent $a$ quantifies the "unbalance" or anisotropy of the data; a larger $a$ indicates a steeper decay in variance across directions.

The study employs a rigorous mathematical framework involving:

  • Gradient Flow Analysis: Modeling optimization as a continuous-time process.
  • Infinite-Dimensional ODEs: Capturing the complex interactions in anisotropic data that simpler models overlook.
  • Duhamel-Volterra Reduction: A technique used to solve the resulting infinite hierarchy of coupled equations, enabling the derivation of closed-form scaling laws for the Mean Squared Error (MSE).

The Three-Phase Learning Trajectory

A central finding of the research is that learning under anisotropy is not a uniform descent but a structured three-act process:

  1. Phase I (Fast Escape): The model rapidly moves away from its initial state (zero correlation with the signal) by capturing high-variance directions.
  2. Phase II (Macroscopic Convergence): Summary statistics, such as overall alignment and energy, begin to stabilize.
  3. Phase III (Spectral-Tail Learning): The model enters a prolonged phase of learning from the low-variance directions in the spectral tail.
The three-phase learning trajectory illustrated through the evolution of alignment $u(t)$, energy $s(t)$, and MSE.

This trajectory reveals a fundamental Escape-Convergence Trade-off: high anisotropy (large $a$) accelerates the "Fast Escape" phase but significantly slows down the final convergence in the "Spectral-Tail" phase. Essentially, the model identifies the primary signal directions almost instantly but requires vast amounts of time to resolve the fine details hidden in the tail.

Key Results and Scaling Laws

The authors derive explicit scaling laws for the MSE decay, proving that the final rate of convergence is directly dictated by the power-law exponent $a$. Specifically, the MSE follows a power-law behavior $MSE(T_2 + \tau) \approx MSE(T_2) S_d(\tau)$, where $S_d(\tau)$ represents a spectral average.

MSE decay curves for different anisotropy levels $a$. Higher $a$ values lead to faster initial drops but slower long-term convergence.

Numerical simulations using synthetic anisotropic Gaussian inputs and Stochastic Gradient Descent (SGD) confirm these theoretical predictions. The experiments demonstrate that as the spectral decay $a$ increases, the error curves flatten out in the late stages of training, precisely as predicted by the derived equations.

Conclusion and Future Directions

This work provides a first-principles explanation for the power-law slopes observed in empirical learning curves. By showing how data "shape" influences optimization speed and precision, it moves theoretical understanding closer to the realities of modern machine learning.

Future research directions include relaxing Gaussian assumptions to extend the proofs to broader data distributions, formalizing the dynamics of discrete SGD steps, and applying these anisotropic scaling laws to multi-layer neural networks to fully capture the complexity of deep learning practice.

Summary updated

BibTeX
@inproceedings{braun2026fast,
  author = {Guillaume Braun and Bruno Loureiro and Ha Quang Minh and Masaaki Imaizumi},
  title = {{Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Ae4eZpkXBX},
}
Figure