Abstract
Recently, it has been shown that optimization problems in both infinite and finite dimensions can display isostatic phases. In these phases, local minima of the cost function are marginally stable configurations characterized by non-linear excitations. This work introduces an athermal adiabatic algorithm to explore these rough high-dimensional landscapes continuously. Focusing on the spherical perceptron with a linear cost function (hinge loss), we investigate the statistics of avalanches triggered when local minima are destabilized. We find that plastic rearrangements are power-law distributed and characterize the critical properties of the unjamming transition, revealing logarithmic scaling in energy and pressure.
Introduction
In high-dimensional optimization, common in physics and machine learning, the landscape of the cost function often dictates the efficiency of search algorithms. A particularly intriguing regime is the "isostatic" phase, where the number of active constraints exactly matches the number of degrees of freedom. In this phase, local minima are not merely sub-optimal; they are marginally stable. A vanishingly small perturbation can trigger a cascade of rearrangements, or "avalanches," similar to those observed in granular materials and structural glasses. While theoretical frameworks have predicted these phases, continuous exploration of such landscapes has remained a challenge.
The Athermal Adiabatic Algorithm
To address this, we developed an "athermal adiabatic" algorithm. Unlike standard gradient descent, which may jump between disconnected minima, our approach follows a local minimum as a control parameter—pressure $p$—is varied. By treating optimization as a physical process of compression, we can drive the system through its landscape.
The algorithm utilizes a Lagrangian formulation where Lagrange multipliers represent the forces exerted by satisfied constraints. A configuration becomes unstable when a "contact force" leaves its stability interval. At this point, the algorithm simulates a "plastic event"—a sequence of rearrangements—until a new stable isostatic minimum is reached. To find the initial starting point at the jamming transition, we use a smoothed version of the Lagrangian to produce a valid configuration. This "surfing" technique allows for the precise study of transition statistics and critical behavior near the unjamming point.
Results and Discussion
The phase diagram of the spherical perceptron with hinge loss reveals a distinct "Critical Phase" where the landscape is non-convex and isostatic ($c=1$). In this region, the system exhibits marginal stability.
Our simulations across various system sizes ($N=32$ to $2048$) show that the rearrangements triggered by instability are scale-free. The avalanche sizes follow a power-law distribution with a critical exponent $\tau \approx 1.41$, a value consistent with observations in sheared hard spheres. This suggests a deep universality between optimization landscapes and the physics of glasses.
Furthermore, we investigated the force distributions and pressure scaling at the jamming point. The distribution of forces $\rho(f/p)$ displays a characteristic pseudogap, and the pressure at first overlap $p_J(N)$ follows a logarithmic scaling law $\propto 1/\sqrt{\log N}$. These logarithmic corrections are a direct consequence of the linear interaction potential (hinge loss) and distinguish this system from those with harmonic potentials.
Conclusion
By treating optimization as a physical process, we have uncovered the fractal nature of isostatic landscapes. The discovery of scale-free avalanches and specific logarithmic scaling laws near the unjamming transition provides a new lens through which to view high-dimensional optimization. Future work will explore the applicability of these findings to more complex architectures, such as deep neural networks, and investigate whether "annealing" through compression-decompression cycles can identify deeper, more stable minima.
Summary updated