Theoretical deep learning often faces a daunting challenge: analyzing the optimization landscape of a deep neural network is notoriously difficult due to the highly non-linear, nested composition of its layers. To bypass this complexity and make mathematical analysis tractable, researchers introduced a powerful simplification known as the Unconstrained Features Model (UFM).
The core intuition of the UFM is to treat the activations of the network's penultimate layer as completely free, independent optimization variables. Instead of modeling how weights in earlier layers transform the input data step-by-step, the UFM assumes the network's backbone is infinitely expressive—capable of mapping any input to any arbitrary point in the embedding space. This mathematical "hack" decouples the final classification layer from the architectural constraints of the preceding layers, transforming a deeply nested optimization problem into a much simpler matrix optimization task.
The Geometric Reward of Unconstrained Features
By adopting the unconstrained feature assumption, theorists unlocked a elegant explanation for how deep networks organize information at the end of training. Under the UFM, researchers proved that networks naturally gravitate toward a state of perfect geometric simplicity known as Neural Collapse (NC).
In this collapsed state, the internal representations of data points belonging to the same class converge to a single point (their class mean). Furthermore, the vectors representing these class means arrange themselves into a Simplex Equiangular Tight Frame (ETF)—a perfectly symmetrical structure where every class vector is as far apart from every other class vector as geometrically possible, perfectly aligned with the final layer's classifier weights.
Using the UFM framework, Jinxin Zhou et al. demonstrated in On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features that the optimization landscape under Mean Squared Error (MSE) loss is remarkably benign, containing no bad local minima. Similarly, Wenlong Ji et al. showed in An Unconstrained Layer-Peeled Perspective on Neural Collapse that this convergence to symmetric geometry is an inherent property of gradient descent under cross-entropy loss, even without explicit regularization or weight decay. The UFM also allowed Christos Thrampoulidis et al. to map the benign landscape of self-supervised objectives in Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features, proving that contrastive embeddings also collapse into ideal geometric configurations.
The Abstraction Gap: Why Features Aren't Free
While the UFM provided a breakthrough in tractability, it introduced a significant abstraction gap. In a real-world deep network, penultimate features are not free variables; they are explicitly bound by the topology of the network and the distribution of the input data.
By treating features as unconstrained, the UFM overlooks several critical realities of deep learning:
- Architectural Inductive Biases: Real networks use specific structural mechanisms—such as convolutions, residual connections, and attention layers—that restrict the space of reachable feature representations. As noted by Chong You et al. in Are All Losses Created Equal: A Neural Collapse Perspective, it remains unclear under pure UFM theory how these distinct architectural designs interact with or potentially prevent total geometric collapse.
- Data Independence: The UFM assumes features can be optimized completely independently of the input data. However, Joan Bruna et al. argued in Extended Unconstrained Features Model for Exploring Deep Neural Collapse that this fails to capture how real-world data distributions dictate feature arrangements. Furthermore, Vignesh Kothapalli et al. demonstrated in Can Kernel Methods Explain How the Data Affects Neural Collapse? that standard data-independent theoretical tools, like the Neural Tangent Kernel (NTK), fail to explain how specific input data properties (such as sample size and dimensionality) drive representation simplification.
- Layer-Wise Dynamics: Neural Collapse is not an instantaneous phenomenon confined to the final layer; it develops progressively throughout the network hierarchy. By collapsing the entire network into a single layer of free variables, the UFM fails to model how depth itself drives feature simplification.
How Real Architectures Constrain Representations
Recent work highlights that a network's architecture acts as a geometric filter, shaping what the model can "see" and represent. For example, Jeff Z. HaoChen et al. proved in A theoretical study of inductive biases in contrastive learning that a model's architecture forces it to focus exclusively on "minimal implementable clusters" that its specific design can represent, ignoring complex spurious structures in the data geometry.
Similarly, in language modeling, Yize Zhao et al. discovered a phenomenon called "subspace-collapse" in Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations. They showed that the sparsity patterns of language data interact with attention and normalization layers to align contexts with identical next-token distributions in the embedding space—a structural interaction that abstracting away the architecture would completely obscure.
To capture the progressive nature of this simplification, Tom Tirer et al. introduced a perturbation-based framework in Perturbation Analysis of Neural Collapse (Correction: Perturbation Analysis of Neural Collapse). By constraining features to remain close to the outputs of their preceding layers, the authors provided the first formal proof of why within-class variability decreases layer-by-layer, directly linking architectural depth to geometric collapse.
Bridging the Gap: Constrained Feature Theory
To resolve the disconnect between mathematical tractability and practical deep learning, a major wave of recent research has focused on proving Neural Collapse and benign optimization landscapes without relying on the unconstrained feature assumption. Instead of optimizing over free variables, these works optimize directly over the actual network parameters of multi-layer architectures.
Deep ResNets and Transformers
The authors of Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers successfully bridged the gap for production-grade models. Peter Súkeník et al. proved that Neural Collapse is the mathematically optimal outcome for deep ResNets and Transformers, demonstrating that the structural constraints and normalization layers used in practice do not prevent collapse, but rather cause the geometric organization to tighten as networks grow deeper.
Shallow ReLU Networks
Moving beyond unconstrained proxies to explicitly model non-linearities, Hancheng Min et al. provided a rigorous analysis in Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data. They demonstrated that when optimizing over actual network weights under gradient flow, the interaction between the ReLU non-linearity and the data structure naturally drives the network toward low-dimensional, collapsed representations.
Wide Networks and Kernel Alignment
Rather than treating penultimate features as independent variables, Arthur Jacot et al. proved in Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse that wide neural networks trained with weight decay naturally exhibit Neural Collapse. This line of work connects feature simplification to the network's internal gradient dynamics. As Mariia Seleznova et al. showed in Neural (Tangent Kernel) Collapse, when a network's Neural Tangent Kernel aligns with class labels during training, this gradient alignment naturally forces the underlying multi-layer architecture to collapse its internal representations.
By moving past the unconstrained feature assumption, deep learning theory is successfully transitioning from idealized proxies to realistic, architecture-aware frameworks. This shift allows researchers to understand not just the geometric destination of training, but how the interplay of data, depth, and architectural constraints drives models there.
Key Foundational Work
A Geometric Analysis of Neural Collapse with Unconstrained Features (2021) — contributed 2 concept elements to this direction.
On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers (2021) — contributed 1 concept elements to this direction.
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks (2020) — contributed 1 concept elements to this direction.