TL;DR
Neural network training transitions from a highly sensitive, chaotic exploration phase to a stable refinement phase. By analyzing training trajectories through a novel "interval-wise" lens, this paper discovers the Chaos Effect (extreme early sensitivity to tiny parameter tweaks) and the Cone Effect (late-stage functional confinement within a narrow angular region), providing a structural and dynamical map of how deep networks actually learn.
What Problem Does This Paper Address?
While it is widely accepted that deep neural networks undergo a phase transition during training—shifting from rapid adaptation to stabilization—fully characterizing this transition remains an open challenge.
Most existing studies analyze training dynamics from a "point-wise" perspective, taking static snapshots of network properties at isolated time steps. This approach fails to capture the continuous temporal evolution of learning. The authors address this gap by introducing an interval-wise framework that compares network states across different time windows, revealing hidden structural shifts in how the model navigates the loss landscape.
Methods
The authors propose an interval-wise analysis framework and design two primary experimental setups to uncover the two phases of learning:
- Perturbation Analysis (The Chaos Effect): To test the network's sensitivity to initial conditions, the authors train two identical networks using the exact same stochastic gradient descent (SGD) noise. At a specific iteration ($t_0$), they inject an imperceptibly small perturbation ($\|\epsilon\|_0 = 10^{-7}$) into the parameters of one network. They then track how much the two trajectories diverge at a later time ($t_1$) by measuring parameter dissimilarity, loss barriers, and prediction disagreement rates.
- eNTK Tracking (The Cone Effect): The authors track the evolution of the empirical Neural Tangent Kernel (eNTK)—a matrix representing how the network's outputs change relative to its parameters. By calculating the kernel distance between different training steps, they map out how the network's underlying function space representation shifts over time.
- Linearized Switching: To determine whether the stable phase behaves like a simple linear model (the "lazy training" regime), they implement a hybrid training method: training standardly up to iteration $t$, then switching to linearized training up to the final iteration.
Datasets
The experiments are conducted on standard image classification benchmarks:
- CIFAR-10 trained using VGG-16 and ResNet-20 architectures.
No new benchmarks or datasets were introduced in this work.
Key Insights and Ideas
- The Chaos Effect is Real: Early training behaves like a chaotic physical system. A tiny tweak to the weights before a critical "inflection point" causes the network to diverge into a completely different valley of the loss landscape, resulting in high loss barriers and entirely different test predictions. Tweaking the weights after this point has virtually no impact.
- The Cone Effect Restricts Function Space: After passing the chaotic phase, the network's eNTK does not freeze (which disproves pure "lazy training"), but its evolution becomes tightly constrained. The functional trajectory gets trapped within a narrow, cone-shaped angular region.
- The Necessity of Late-Stage Non-Linearity: Even though the second phase is highly stable and constrained, it is not purely linear. The authors show that switching to a completely linearized training mode too early harms final test accuracy, proving that the bounded non-linear evolution within the "cone" is vital for generalization.
Key Results
- Sharp Inflection Points: For VGG-16 on CIFAR-10, the critical inflection point occurs sharply around iteration 2,500. Perturbations introduced before this mark lead to a test disagreement rate of up to 30–50% later in training. For ResNet-20, this transition happens much earlier, between iterations 100 and 500.
- eNTK Stabilization: Kernel distance measurements confirm that after the inflection point, the distance between the eNTK at the current iterate $t$ and a reference point $\tau$ stabilizes and remains nearly constant, visually forming a narrow cone in the trajectory plots.
- Generalization Advantage: In the switching experiments, final test performance steadily improves the later the network is allowed to stay in its standard non-linear training mode before switching to linearized training, particularly showing a steep performance jump once training passes the 2,000-iteration mark.
Limitations and Future Work
- Lack of Theoretical Proof: The paper is primarily an empirical study. While it provides strong experimental evidence for the Chaos and Cone effects, a rigorous mathematical framework explaining why these boundaries emerge naturally during SGD is left for future work.
- Limited Domain Scope: The experiments focus exclusively on image classification tasks using convolutional architectures (VGG and ResNet). The authors note that exploring whether these interval-wise dynamics hold true for other architectures (like Transformers) or domains (like Large Language Models and Reinforcement Learning) is an important next step.
Summary updated