To make deep learning theory mathematically tractable, researchers frequently rely on a powerful simplification: they treat a neural network's feature extractor as an omnipotent, unconstrained engine. Under this unconstrained feature assumption (often called the infinite encoder assumption), a model is assumed to possess the absolute freedom to map any input data point to any arbitrary location in the embedding space.
This mathematical luxury allows theorists to decouple the optimization of a loss function from the messy realities of backpropagation, layer architectures, and parameter counts. By assuming the network has infinite capacity to position its activations, researchers can solve for the ideal geometric configurations that minimize a given loss. However, a growing body of work reveals a significant gap between these elegant theoretical global minima and the practical performance of finite-capacity architectures.
The Elegance of Infinite Capacity
Why do theorists love the unconstrained feature assumption? Because it yields beautiful, closed-form geometric solutions that help explain why certain loss functions perform well.
Consider the phenomenon of Neural Collapse, where the final-layer features of a trained network collapse into the vertices of a regular simplex (like an equilateral triangle in 2D or a tetrahedron in 3D). As shown by Graf et al., assuming an infinitely powerful encoder allows one to prove that both Cross-Entropy and Supervised Contrastive losses force class representations into this exact same symmetric geometry at minimal loss.
This elegant symmetry extends even to complex multi-task and self-supervised settings:
- Multi-Task Learning: Wang et al. demonstrated that under the unconstrained feature assumption, classifiers for different tasks don't just collapse individually; they arrange themselves into perfectly orthogonal subspaces, allowing a single shared feature vector to serve multiple specialized heads simultaneously.
- Self-Supervised Learning: Luthra et al. leveraged the assumption to prove a rigorous theoretical bridge showing that self-supervised contrastive learning acts as a "negatives-only" version of supervised learning, explaining why label-agnostic models produce high-quality semantic clusters.
In all these cases, the assumption works because it turns a highly non-convex optimization problem over network parameters into a clean, convex geometric problem over the embedding space itself.
Where Architecture Fights Back
While mathematically convenient, the fiction of the infinite encoder creates a stark disconnect with reality. Real-world networks are tightly bounded by their architecture. They have finite widths, limited depths, specific activation functions, and inherent inductive biases that prevent embeddings from moving arbitrarily. When these constraints are introduced, the beautiful geometric proofs begin to fracture.
For example, Chen et al. pointed out that the infinite encoder assumption fails to predict "class collapse"—a destructive failure mode in contrastive learning where all images in a class map to a single point, erasing fine-grained subclass details. Because real networks have architectural bottlenecks, they cannot perfectly balance spreading out representations while preserving internal structures without explicit algorithmic help.
Furthermore, minor architectural choices can completely alter the optimization landscape:
- The Role of Activations: Kini et al. demonstrated that simply adding a ReLU activation to the final layer of a network prevents class imbalances from distorting internal representations. This is a nuance completely missed by unconstrained models, which assume the network can map inputs anywhere without being blocked by a non-negativity constraint.
- The Impact of Regularization and Augmentation: When networks operate with finite capacity, data augmentations like Mixup do not merely cause linear blending. Instead, Fisher et al. discovered that Mixup forces features into intricate "channels" that hug decision boundaries. This sophisticated geometry is born entirely out of the tension between the augmentation constraints and the network's finite capacity.
Ideal Theory (Unconstrained Features) Real-World Architecture (Constrained)
Class A Class A
○ ○
/ \ / \ <- Restricted by
/ \ / \ ReLU / Width
/ \ ○-----○
○-------○ Class B Class C
Class B Class C (Distorted by bottlenecks)
Expressivity vs. Learnability
The tension between ideal theory and practical constraints also undermines classical approximation frameworks. For instance, "feature geometry" frameworks developed by Xu et al. and Xu et al. treat statistical dependencies as vectors in a function space, showing how networks can extract optimal, orthogonal features. Yet, as the authors acknowledge, these guarantees presuppose universal approximation capabilities. In practice, smaller or highly constrained architectures lack the expressive power required to satisfy these assumptions, leading to suboptimal feature extraction.
This reveals a deeper flaw in traditional neural network approximation theory. For decades, theorems proved that networks could approximate almost any function given infinite parameters or infinite precision. However, as Wei et al. and Wei et al. argue, these proofs "cheat" by ignoring whether a network can actually learn these functions from a finite dataset.
To bridge this gap, the authors introduced the framework of Statistically Meaningful (SM) Approximation. Instead of asking what an infinitely large network can represent, SM approximation requires that the approximating network must also possess low sample complexity. Under this realistic lens, the authors proved that overparameterized Transformers can efficiently simulate complex systems like bounded Turing machines with a sample complexity that scales only logarithmically with computation time—shifting the focus from raw, unconstrained expressivity to practical learnability.
Bridging the Gap
The empirical evidence across various architectures confirms that theoretical expressivity does not equal practical learnability. Uteuliyeva et al. showed that while Fourier Neural Networks possess superior theoretical capabilities in approximating synthetic functions compared to standard networks, they fail to outperform simple sigmoid-activated networks on real-world tasks like image recognition. Conversely, Takhanov demonstrated that practical, finite-width networks trained via gradient descent can effectively learn functions that appear theoretically challenging under classical frameworks like Barron's theorem.
To build a truly predictive theory of deep learning, the field is moving away from the fiction of the unconstrained encoder. Bridging this gap requires constructive training methods, such as those proposed by Lin et al., that are simultaneously theoretically sound and practically efficient under real-world architectural constraints. Only by accounting for finite capacity, activation bottlenecks, and statistical sample complexity can we understand why neural networks work in practice, not just in an idealized geometric sandbox.
Key Foundational Work
Fourier neural networks: A comparative study (2020) — contributed 2 concept elements to this direction.
Constructive Neural Network Learning (2018) — contributed 1 concept elements to this direction.