Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets

Arthur Jacot, Alexandre Kaiser

TL;DR

By reformulating continuous-depth "Leaky ResNets" through the lens of classical mechanics, this paper reveals that feature learning is driven by a tug-of-war between kinetic energy (penalizing rapid changes) and potential energy (a "Cost of Identity" that penalizes high-dimensional representations). For deep networks, potential energy dominates, forcing representations to rapidly compress into a low-dimensional bottleneck, evolve slowly, and expand again at the output—an insight the authors exploit to design adaptive, non-uniform layer step sizes.

What Problem Does This Paper Address?

Despite the ubiquity of deep neural networks, our theoretical understanding of how and why representations evolve across layers remains incomplete. Recent work has demonstrated that deep networks exhibit an implicit bias toward learning low-rank features (a phenomenon known as the Bottleneck structure), where hidden representations collapse to a minimal intrinsic dimension.

However, existing frameworks like the Information Bottleneck rely on mutual information, which is ill-defined for continuous, deterministic empirical data. Meanwhile, existing continuous-depth models (NeuralODEs) lack an intuitive physical framework that simultaneously explains representation dynamics, depth-dependent time scales, and low-dimensional representation bias.

Methods

The authors study Leaky ResNets, a continuous formulation governed by the ODE: $$\partial_p \alpha_p(x) = -\tilde{L}\alpha_p(x) + W_p \sigma(\alpha_p(x)), \quad p \in [0, 1]$$ where the leakage parameter $\tilde{L}$ serves as an "effective depth," smoothly interpolating between standard ResNets ($\tilde{L} = 0$) and Fully-Connected Networks ($\tilde{L} \to \infty$).

Key innovations include:

  1. Hamiltonian & Lagrangian Reformulation: The optimization over weights with $L_2$-regularization (weight decay) is converted into an optimization over activation paths $A_p$ across the dataset. The Lagrangian cleanly separates into:
  2. Kinetic Energy ($\frac{1}{2\tilde{L}}\|\partial_p A_p\|_{K_p}^2$): Favors minimal representation change between layers.
  3. Potential Energy / Cost of Identity (COI) ($-\frac{\tilde{L}}{2}\|A_p\|_{K_p}^2$): Measures the Frobenius norm of the smallest weight matrix needed to represent the identity mapping. Stable local minima of this cost precisely equal the rank of the activation matrix ($\text{Rank}(A)$).
  4. Conserved Quantities & Timescale Analysis: The associated Hamiltonian $H$ is conserved along the depth $p$, establishing that layers where representation dimension is high must experience high-speed dynamics, whereas low-dimensional layers evolve slowly.
  5. Adaptive Layer Discretization: Guided by this timescale separation, the authors introduce an adaptive step-size scheme $\rho_\ell \propto \|A_\ell - A_{\ell-1}\|^{-1}$ to concentrate discrete layers near the fast boundary transitions rather than uniformly wasting layers in the static bottleneck.

Datasets

The paper evaluates the theory using synthetic teacher-student regression tasks:

  • Teacher networks with explicit low-rank bottlenecks (e.g., compositions mapping $\mathbb{R}^{20} \to \mathbb{R}^5 \to \mathbb{R}^{20}$ with $k^* = 5$, and $\mathbb{R}^{30} \to \mathbb{R}^6 \to \mathbb{R}^3 \to \mathbb{R}^{30}$ with $k^* = 3$).
  • Datasets consist of 1,000 synthetic training points and 1,000 test points, centered and normalized.
  • No real-world image or NLP benchmarks are tested in this paper.

Key Insights and Ideas

  • Energy Conservation in Deep Learning: Because the Hamiltonian $H$ is invariant across layers $p$, the identity $-\frac{2}{\tilde{L}}H \approx k^*$ directly tracks the intrinsic bottleneck rank $k^*$ of the task.
  • The Physical Origin of Bottlenecks: For large effective depth $\tilde{L}$, potential energy dominates. The network incurs a huge penalty for remaining high-dimensional. Consequently, it executes a fast "jump" from high-dimensional inputs to a low-dimensional manifold, drifts along low-cost representation geodesics, and jumps back to the output dimension at the very end.
  • Identities Are Expensive in High Dimensions: The "Cost of Identity" mathematically formalizes why networks compress: holding information across layers in high dimensions requires significantly larger parameter norms than doing so in low dimensions.
  • Uniform Depth is Suboptimal: Standard ResNets space layers uniformly ($\rho_\ell = 1/L$). Because representation velocity varies by orders of magnitude across depth, uniform spacing undersamples fast boundary layers and oversamples the stationary bottleneck.

Key Results

  • Recovery of True Rank: Across varying effective depths ($\tilde{L} \in [2, 7.5]$), the normalized Hamiltonian $-\frac{2}{\tilde{L}}H$ closely converges to the ground-truth bottleneck ranks ($k^* = 5$ and $k^* = 3$).
  • Vanishing Kinetic Energy: In the bottleneck region, representation derivative norms approach zero, confirming slow drift within the low-rank subspace.
  • Improved Discretization: The proposed adaptive step-size scheme consistently achieves lower test error compared to equidistant discretization across depths ranging from $L = 4$ to $L = 22$, with negligible computational overhead (~2% training time increase).

Limitations and Future Work

  • Synthetic Scope: Experiments are limited to low-dimensional synthetic teacher-student problems; validation on standard real-world architectures (e.g., Vision Transformers, ConvNets on ImageNet) remains open.
  • Singularities in Pseudo-Inverses: The exact Cost of Identity involves matrix pseudo-inverses that blow up near low-rank representations. While the authors resolve this using a regularized $\gamma$-relaxation, exact mathematical handling of these boundaries requires further formalization.
  • Absence of Intermediate Jumps: While the theory allows for sudden dimensional jumps in intermediate layers, only boundary jumps were observed experimentally; determining the exact conditions precluding interior jumps is left for future work.

Summary updated

Abstract

We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' hyper-parameter $\tilde{L}$. In the infinite depth limit, we study 'representation geodesics' $A_{p}$: continuous paths in representation space (similar to NeuralODEs) from input $p=0$ to output $p=1$ that minimize the parameter norm of the network. We give a Lagrangian and Hamiltonian reformulation, which highlight the importance of two terms: a kinetic energy which favors small layer derivatives $\partial_{p}A_{p}$ and a potential energy that favors low-dimensional representations, as measured by the 'Cost of Identity'. The balance between these two forces offers an intuitive understanding of feature learning in ResNets. We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work: for large $\tilde{L}$ the potential energy dominates and leads to a separation of timescales, where the representation jumps rapidly from the high dimensional inputs to a low-dimensional representation, move slowly inside the space of low-dimensional representations, before jumping back to the potentially high-dimensional outputs. Inspired by this phenomenon, we train with an adaptive layer step-size to adapt to the separation of timescales.
BibTeX
@inproceedings{jacot2025hamiltonian,
  author = {Arthur Jacot and Alexandre Kaiser},
  title = {{Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets}},
  booktitle = {CPAL (Proceedings Track)},
  year = {2025},
  url = {https://openreview.net/forum?id=C8OsIiVcyC},
}
Figure