Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets

Arthur Jacot, Alexandre Kaiser

Abstract

This paper introduces a physics-inspired Hamiltonian framework to explain how deep neural networks learn features. By analyzing "Leaky ResNets," we demonstrate that optimal feature learning creates a "bottleneck structure" where the network rapidly compresses data into low-dimensional representations to minimize parameter costs. This process leads to a natural separation of timescales across layers, formalizing the evolution of representations as geodesics in a physical system where the Hamiltonian remains constant and directly relates to the intrinsic rank of the task.

Introduction

Despite the massive success of deep learning, a rigorous mathematical description of feature learning—the process by which a network transforms raw input into useful, high-level representations—remains elusive. While empirical observations like "Neural Collapse" or the Information Bottleneck theory provide intuition, they lack a formal dynamical explanation. This work addresses the "Bottleneck" phenomenon: the tendency of deep networks to compress information into a low-dimensional space in the middle layers. We formalize this using the language of classical mechanics, treating the path of data through a network as a "geodesic" (the shortest or most efficient path) in representation space.

Methods: Leaky ResNets and Hamiltonian Dynamics

We study Leaky ResNets, a variant of Residual Networks that can interpolate between standard ResNets and Fully-Connected Networks (FCNNs) via an "effective depth" parameter ($\tilde{L}$). In the infinite-depth limit, the network is modeled as a continuous path $A_p$ from input to output.

The optimization problem of minimizing weight norms is mapped to a physical system defined by two primary components:

  • Kinetic Energy: Favors smooth changes between layers, penalizing rapid derivatives in the representation path.
  • Potential Energy (Cost of Identity, COI): A novel metric measuring the "cost" of representing the identity mapping. Crucially, this cost is lower for low-dimensional representations, creating a mathematical bias toward compression.
Evolution of singular values (top) and energy components (bottom) across network layers. The Hamiltonian $-\frac{2}{L}\mathcal{H}$ remains constant and aligns with the target rank $k^*$.

Key Insights: The Physics of Feature Learning

A central discovery of this framework is that the Hamiltonian is a constant of motion. Just as energy is conserved in a closed physical system, a specific Hamiltonian value remains constant across all layers of an optimized network. This constant provides a remarkably accurate estimate of the "Bottleneck Rank" of the task.

Furthermore, the inverse of the representation's covariance matrix ($K_p^+$) acts as the "mass matrix" of the system, determining how much "force" is needed to change features at any given layer. In very deep networks, the potential energy dominates, causing a "separation of timescales": the network moves extremely fast at the beginning (compressing input) and end (expanding to output), but moves slowly through the low-dimensional bottleneck in the middle.

Experimental Results

Using a teacher-student setup with synthetic datasets of known intrinsic rank ($k^*$), we validated the theoretical predictions of the Hamiltonian framework.

Rank Recovery

The student networks successfully recovered the true rank of the teacher networks. As shown in Figure 2, the Hamiltonian constant $- \frac{2}{\tilde{L}} \mathcal{H}$ provides an increasingly accurate estimate of $k^*$ as the effective depth of the network increases.

The Hamiltonian constant $- \frac{2}{\tilde{L}} \mathcal{H}$ converges to the true intrinsic rank $k^*$ as the effective depth $\tilde{L}$ increases.

Bottleneck Emergence and Adaptive Training

Visualizations of weight matrix singular values confirm that for large effective depths, the network maintains a stable, low-rank representation for the majority of its layers. Inspired by the "jump" behavior at the boundaries of the geodesic, we proposed an Adaptive Discretization scheme. By placing more layers where representations change most rapidly (at the ends of the network), we achieved consistently lower test errors compared to standard equidistant layer spacing.

Conclusion and Future Work

The Hamiltonian framework offers a robust theoretical foundation for understanding the emergence of bottlenecks in deep learning. While currently focused on synthetic tasks, this approach provides a path toward analyzing large-scale models. Future research will address the numerical stability of the "Cost of Identity" and explore these dynamics in real-world architectures like Transformers and LLM embeddings.

Summary updated

BibTeX
@inproceedings{jacot2025hamiltonian,
  author = {Arthur Jacot and Alexandre Kaiser},
  title = {{Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets}},
  booktitle = {CPAL (Proceedings Track)},
  year = {2025},
  url = {https://openreview.net/forum?id=C8OsIiVcyC},
}
Figure