In machine learning, we often visualize optimization as a ball rolling down a hill into a stable valley. We assume that if we design a good loss function, there must exist a perfect, finite set of optimal weights $\theta^*$ at the bottom of that valley.
But what if the valley has no bottom? For many foundational settings—such as classification with separable data, cross-entropy loss, or networks with smooth activation functions—the loss function can only be minimized by driving the model's weights toward infinity. This phenomenon, known as parameter blow-up or weight divergence, means that a true global minimum does not exist within finite space; instead, the optimization problem only admits an infimum.
Understanding why weights blow up, how they behave at infinity, and how this affects a model's ability to generalize has become a crucial frontier in deep learning theory.
The Mechanisms of Parameter Blow-Up
Why would an optimization algorithm continuously push weights toward infinity? This behavior typically stems from two distinct sources: the geometry of classification losses on separable data, and the mathematical limits of smooth activation functions.
1. Data Separability and Strict Loss Functions
Consider a binary classification problem where the training data is linearly separable. If we train a linear model using a loss function that asymptotically approaches zero—such as the logistic or exponential loss—the loss can never truly equal zero for any finite weight vector.
To minimize the loss, the model must maximize its confidence on every training example. As Michael Collins et al. noted in their early unification of boosting and logistic regression, this forces the magnitude of the weights to grow indefinitely. Matus Telgarsky et al. later emphasized that this reality directly contradicts standard statistical learning assumptions, which traditionally require weight vectors to remain within a bounded space to guarantee stability.
2. Smooth Activation Functions and Function Approximation
Parameter blow-up is not exclusive to linear models or classification losses; it also occurs due to the choice of activation functions in deep networks. When a network uses smooth, bounded activations like sigmoid or tanh, it may be mathematically impossible to perfectly fit certain target functions using finite weights.
Arnulf Jentzen et al. demonstrated that in these scenarios, the risk landscape completely lacks a global minimum at any finite coordinate. To reduce error, gradient descent must continuously increase the weight magnitudes to make the activation curves steeper, effectively trying to mimic step functions. Julian Kranz et al. formalized this behavior, proving a fundamental dichotomy: gradient flow on these landscapes either converges to a finite point or diverges to infinity while achieving asymptotic optimality.
Standard Optimization:
Loss Landscape: \___/ --> Weights converge to a stable, finite minimum θ*
Unbounded Optimization:
Loss Landscape: \___ --> Weights grow endlessly toward infinity to minimize loss
The Theoretical Crisis of Unattainability
When an optimization problem lacks a finite minimum, standard mathematical guarantees collapse. Most classical convergence proofs rely on the assumption that the optimization path remains within a compact, bounded region.
As Quoc-Tung Le et al. highlighted, when an optimizer chases an infimum at infinity, the lack of weight stability can cause severe numerical instability and training failure. This creates a major hurdle for modern distributed systems; for example, Hossein Taheri et al. showed that weight divergence invalidates traditional theoretical frameworks for decentralized learning on separable datasets, necessitating entirely new tools to analyze how these models generalize.
To address this crisis, theorists have had to reinvent how they calculate convergence rates. Instead of measuring the distance to a fixed optimal point $\theta^*$, they must measure how quickly the loss decreases as the weights escape to infinity. Matus Telgarsky provided a breakthrough in this area by decomposing general boosting problems into "attainable" and "weakly learnable" components, proving that fast convergence rates are still provably guaranteed even when the optimal weights are unattainable.
Order at Infinity: Directional Convergence and Implicit Bias
If the magnitude of a network's weights grows endlessly, does the model's behavior become completely chaotic and unpredictable? Remarkably, the answer is no. While the length of the weight vector blows up, its direction stabilizes.
Matus Telgarsky proved that for a wide class of deep networks trained with classification losses, the weights eventually achieve directional convergence and alignment (Directional convergence and alignment in deep learning). This means that even though the weights grow infinitely large, the model's decision boundaries and prediction surfaces settle into a perfectly stable state. Even when the data is nonseparable, Matus Telgarsky demonstrated that the diverging weight vector tracks a mathematically precise "optimal ray" that optimally balances the separable and overlapping portions of the dataset.
This directional growth is guided by a powerful force known as implicit bias. The weights do not escape to infinity in a random direction; they are biased toward solutions that maximize the classification margin.
- Gradient Descent: Miroslav Dudík et al. proved that for general strictly decreasing losses, the asymptotic direction chosen by gradient descent aligns perfectly with the explicit $\ell_2$-regularized regularization path. This explains why unregularized gradient descent naturally produces models with excellent generalization properties.
- Mirror Flow: This geometric structure holds across different optimizer geometries. Scott Pesme et al. analyzed mirror flow—the continuous-time variant of mirror descent—showing that as it minimizes loss "at infinity," it converges toward a specific maximum-margin solution determined by the horizon function of its potential.
Architectural and Algorithmic Countermeasures
Armed with the understanding that optimization often operates at infinity, researchers have identified specific architectural choices and algorithmic properties that can either prevent parameter blow-up or safely accelerate training despite it.
Architectural Safeguards
The choice of network architecture plays a massive role in whether a finite minimum exists. While smooth activations force weights to diverge to fit sharp boundaries, changing the activation function can alter the landscape entirely. Steffen Dereich et al. proved that for shallow residual networks, using ReLU activations guarantees the existence of a global minimizer within a finite space, completely preventing parameter blow-up.
For deeper networks where classification losses still drive weights to infinity, structural components like skip connections and normalization layers become vital. Lachlan E. MacDonald et al. demonstrated that these layers regularize the loss landscape by shifting up the singular values of layer Jacobians. This allows the network to maintain regularity and achieve global convergence even as cross-entropy loss drives the parameters outward.
Exploiting the Geometry of Infinity
Rather than fearing weight growth, some algorithms exploit it to achieve faster training. In standard optimization, using an excessively large stepsize causes gradient descent to diverge catastrophically. However, when optimizing over separable data where the loss landscape flattens out at infinity, the rules change.
Han Bao et al. revealed that under Fenchel–Young losses (such as those based on Tsallis and Rényi entropies), gradient descent can safely operate with arbitrarily large stepsizes. Because the loss landscape becomes highly predictable as the weights grow, the algorithm does not destabilize; instead, it exploits the unbounded space to achieve significantly faster convergence rates ($O(\epsilon^{-1/2})$ or $O(\epsilon^{-1/3})$) than the standard $O(\epsilon^{-1})$ rate typically seen in bounded optimization.
Key Foundational Work
The Fast Convergence of Boosting (2011) — contributed 1 concept elements to this direction.
Logistic Regression, AdaBoost and Bregman Distances (2002) — contributed 1 concept elements to this direction.