Abstract
This report explores the phenomenon of "jamming" criticality within high-dimensional non-convex optimization problems characterized by non-analytic (kinked) cost functions. By analyzing a piecewise-linear perceptron model, we demonstrate that such systems self-organize into a "globally isostatic" state. In this state, constraints arising from multiple kinks collectively balance to maintain marginal stability, exhibiting universal critical exponents identical to those found in hard-sphere jamming.
Introduction
In the physics of granular media, "jamming" describes the transition where a collection of particles becomes rigid. At this critical point, the system is isostatic: the number of constraints (contacts) exactly matches the number of degrees of freedom. This state is marginally stable, where infinitesimal perturbations can lead to large-scale structural reorganizations known as avalanches.
Recent theoretical work has extended this concept to high-dimensional optimization, particularly in neural network models like the perceptron, when the objective function contains non-analytic points or "kinks." While previous studies focused on single-kink potentials, this paper investigates the behavior of systems with multiple kinks. We address whether isostaticity persists as a global property and how "forces" (Lagrange multipliers) and "gaps" (distances to constraint violations) distribute across multiple non-analyticities.
Methodology
The study utilizes the spherical perceptron, a mean-field model for high-dimensional optimization. The model consists of an $N$-dimensional vector $x$ constrained to a sphere, subject to $M = \alpha N$ random constraints.
The primary innovation is the use of a piecewise-linear cost function featuring two kinks at $h=0$ and $h=-H_0$. To explore the landscape of local minima in the non-convex "glassy" phase, the authors employed specialized gradient-descent algorithms. Theoretical stability was analyzed using a Lagrangian framework to define contact forces for each kink and evaluating the Hessian matrix of the system. Synthetic datasets comprised $M$ random $N$-dimensional vectors drawn from a Gaussian distribution, with $N$ ranging from 256 to 2048 and $\alpha=4$ to ensure a glassy regime.
Global Isostaticity and Self-Organization
A central finding of this research is that isostaticity is a global property. In a single-kink model, isostaticity requires $N$ contacts at that specific kink. In the multi-kink piecewise-linear model, the system distributes its $N$ degrees of freedom across all available kinks. Specifically, if $c_0$ and $c_{H_0}$ represent the fraction of contacts at the first and second kinks respectively, the results show that $c_0 + c_{H_0} = 1$ (normalized by $N$).
This self-organization into a marginally stable state is not restricted to a single transition point but persists throughout the jammed glassy phase. The presence of multiple kinks leads to a proliferation of non-linear excitations, where perturbations can trigger contacts to open or close at any kink location, resulting in complex, system-spanning avalanches.
Universality of Critical Exponents
Despite the increased complexity of multiple kinks, the system maintains the same critical exponents observed in hard-sphere jamming. The distribution of gaps $\rho(h)$ displays power-law divergences at each kink, while the force distribution shows "pseudogaps" near the edges of stability.
The gap distribution follows a power law with an exponent $\gamma \approx 0.41$, and the force distribution follows $\theta \approx 0.42$. These results suggest that jamming criticality represents a robust universality class for a wide range of non-convex optimization problems.
Conclusion and Future Directions
This work demonstrates that marginal stability and isostaticity are fundamental features of optimization in landscapes with non-analyticities. While the perceptron is a mean-field model, these findings likely extend to finite-dimensional systems. Future research could explore these dynamics in deeper architectures, such as multi-layer networks with ReLU activation functions, to understand how non-linearities contribute to the stability and efficiency of deep learning models.
Summary updated