Singular Perturbation in Heavy Ball Dynamics
Abstract
Given a $C^{1,1}_\mathrm{loc}$ lower bounded function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ definable in an o-minimal structure on the real field, we show that the singular perturbation $ε\searrow 0$ in the heavy ball system \begin{equation} \label{eq:P_eps} \tag{$P_ε$} ε\ddot{x}_ε(t) + γ\dot{x}_ε(t) + \nabla f(x_ε(t)) = 0, ~~~ \forall t \geqslant 0, ~~~ x_ε(0) = x_0, ~~~ \dot{x}_ε(0) = \dot{x}_0, \end{equation} preserves boundedness of solutions, where $γ>0$ is the friction and $(x_0,\dot{x}_0) \in \mathbb{R}^n \times \mathbb{R}^n$ is the initial condition. This complements the work of Attouch, Goudou, and Redont which deals with finite time horizons. In other words, this work studies the asymptotic behavior of a ball rolling on a surface subject to gravitation and friction, without assuming convexity nor coercivity.
The Heavy Ball System and the Singular Limit
In optimization and physical modeling, the "heavy ball with friction" system is a classic second-order dynamical system used to find the minima of a function $f(x)$. It describes the motion of a particle with mass $\epsilon$ moving through a landscape $f$ under the influence of gravity and a friction coefficient $\gamma$. Mathematically, it is expressed as: $$\epsilon\ddot{x}_\epsilon(t) + \gamma\dot{x}_\epsilon(t) + \nabla f(x_\epsilon(t)) = 0$$ As the mass $\epsilon$ approaches zero, the system is expected to simplify into a first-order gradient flow: $$\gamma\dot{x}(t) + \nabla f(x(t)) = 0$$ This transition is known as a "singular perturbation" because the order of the differential equation changes from two to one, fundamentally altering the nature of the dynamics.
Bridging the Gap to Infinite Time
While previous research established that the heavy ball system converges to the gradient flow over any finite time interval $[0, T]$, the behavior over an infinite time horizon $[0, \infty)$ remained an open question for general functions. Specifically, without the assumptions of convexity (where the landscape is a simple bowl) or coercivity (where the landscape goes to infinity in all directions), it was unknown if a vanishingly small mass could cause a trajectory to escape to infinity, even if the corresponding gradient flow remained bounded.
This paper addresses this gap by proving that for a broad class of "tame" functions, the property of boundedness is robust. If the gradient flow stays within a bounded region, the heavy ball system will also stay bounded for all sufficiently small $\epsilon$. This ensures that the addition of a small inertial term does not destabilize the system's long-term behavior.
The Power of Tame Geometry
To handle non-convex and non-coercive landscapes, the authors utilize o-minimal structures. This framework from model theory identifies "tame" functions that avoid pathological behaviors like infinite oscillations (e.g., $\sin(1/x)$ near the origin).
A key tool within this framework is the Kurdyka-Łojasiewicz (KL) inequality. The KL inequality ensures that the trajectory of a gradient-like system has finite length, preventing it from wandering aimlessly or spiraling forever without converging. By combining this with Lyapunov functions—energy-like quantities that decrease along trajectories—the authors can track the system's state and prove global stability.
Key Results and Robustness
The primary contribution is Theorem 1, which establishes that for a $C^{1,1}_{loc}$ lower-bounded definable function, the global solution to the heavy ball system is uniformly bounded for all small $\epsilon$ if and only if the gradient flow solution is bounded.
Furthermore, the authors apply Fenichel’s theory from Geometric Singular Perturbation (GSP) to show that as $\epsilon \to 0$, the heavy ball trajectories converge uniformly to some gradient flow trajectory. Crucially, they note that while the systems stay bounded, they may not converge to the same equilibrium point. A small amount of momentum can carry the ball over a small ridge to a different local minimum than the massless version would reach, but it will still remain within a predictable neighborhood.
Numerical Illustration
The authors illustrate these theoretical findings using the function $f(x, y) = (xy - 1)^2$. This function is particularly challenging because its set of global minima—the hyperbola $xy=1$—is not coercive; the "valley" of minima extends to infinity along the axes.
As seen in the figure, trajectories starting from $(1, -1)$ with different mass parameters $\epsilon$ all remain bounded and eventually settle on the set of global minima. While the gradient flow ($\epsilon=0$) takes a direct path toward the saddle point before turning, systems with higher mass exhibit oscillatory behavior. Despite these oscillations and the non-coercive nature of the landscape, the trajectories never escape to infinity, confirming the robustness of the boundedness property.
Limitations and Outlook
While the paper proves the existence of a stability threshold $\epsilon_0$, it does not provide a general method for calculating its exact value for a given function. Additionally, the results are tied to the assumption of definability in an o-minimal structure. While this covers most practical applications in machine learning and engineering (polynomials, exponentials, etc.), it does exclude certain non-tame functions.
Future work may extend this framework to systems with variable friction coefficients or explore its implications for accelerated optimization algorithms where "momentum" is intentionally added to escape saddle points or speed up convergence in complex, non-convex landscapes.
Summary updated