In the classical view of machine learning, optimization and generalization are two sides of the same coin: by minimizing the error on a training set (Empirical Risk Minimization), we hope to minimize the error on unseen data (Population Risk). However, as neural networks have grown deeper and more over-parameterized, researchers have uncovered a startling reality: the ability to reach a global minimum in the training landscape is often entirely decoupled from the ability to generalize.
This research direction explores why "solving" the optimization problem—reaching zero training loss—is a necessary but insufficient condition for learning, and how the geometry of the path we take to get there determines the final model's quality.
The Mirage of Global Convergence
A significant body of theoretical work has recently proven that for many modern architectures, reaching a global minimum is actually "easy." For instance, Zenan Ling and colleagues demonstrated that Deep Equilibrium Models (DEQs) are guaranteed to converge to a global optimum when sufficiently wide. Similarly, Zhiyan Ding and Raphaël Barboni have provided proofs for the global convergence of ResNets in the mean-field regime.
Even the "messy" non-convexity of ReLU networks appears less daunting than once feared. C. Daniel Freeman and Joan Bruna showed that as networks become over-parameterized, their loss landscapes become "asymptotically connected," meaning there are low-loss paths between almost any two solutions. Daniel Soudry further argued that sub-optimal local minima essentially vanish in multilayer networks.
But there is a catch. As Mahdi Soltanolkotabi and Samet Oymak point out, proving that gradient descent reaches zero training error provides almost no insight into whether that model will work on a test set. In fact, Tal Burla and Roi Livni have shown that even in simple stochastic convex optimization, an Empirical Risk Minimizer (ERM) can perfectly fit training data while failing completely on the broader population because it has essentially "memorized" the sample noise.
The Stability-Convergence Trade-off
If reaching the minimum isn't the goal, perhaps the speed at which we get there matters? Here, the evidence suggests a "no free lunch" theorem. Chi Jin and Bin Yu proved a fundamental trade-off: algorithms that converge faster to a training minimum (like Nesterov’s Acceleration) are inherently less stable and more prone to overfitting than slower, more "stochastic" algorithms.
This suggests that the "noise" in Stochastic Gradient Descent (SGD) isn't just a byproduct of efficiency—it's a feature that enforces generalization. Gregory Schwartzman uses Kolmogorov complexity to argue that SGD's local progress tends to favor "simple" functions, which are more likely to generalize. However, even this information-theoretic progress doesn't guarantee a small generalization gap if the training objective itself is misaligned with the task logic, a problem Zhe Wu identifies as "rote memorization" versus "rule learning."
Curvature and the Goldilocks Zone
A popular intuition in the field is that "flat" minima generalize better than "sharp" ones. The idea is that if the loss surface is flat, small shifts between the training and test distributions won't cause a massive spike in error. Arwen Bradley and colleagues explore this "shift-curvature" relationship, noting that the generalization gap is often just an offset caused by sampling noise.
However, Artem M. Vysogorets and Anna Dawid caution against over-relying on curvature. They deconstructed the "Goldilocks Zone"—regions of high trainability—and found that these regions do not always coincide with regions of high generalization. A model can be initialized in a regime that makes optimization effortless while still leading to a solution that fails on unseen data.
Maintaining Plasticity via Dual Networks
One of the most practical manifestations of this disconnect is the loss of "plasticity"—the tendency of networks to become rigid and unable to learn new information as they converge. Hojoon Lee and the team behind Hare & Tortoise (H&T) address this by splitting the roles of optimization and generalization.
In their framework, a "Hare" network focuses on fast adaptation (optimization), while a "Tortoise" network consolidates stable knowledge (generalization). By periodically resetting the fast-learner to the stable-learner's weights, they effectively "shed" the spurious noise and memorization that the Hare picks up during its rapid descent into training minima.
Toward a Unified Theory
The critique offered by Gitta Kutyniok remains the central challenge for the field: we have theories for expressivity (what networks can represent) and generalization (how they perform), but they are often isolated from the messy reality of optimization.
Whether through Singular Learning Theory, which accounts for the geometric singularities in parameter space (Susan Wei), or through Ergodic Theory, which views weights as a stable distribution rather than a single point (Haochuan Li), the goal is the same: to understand the hidden bridge between the training loss we can see and the test performance we can only hope for.