When designing deep neural networks, our instinct is often to scale upward and outward. We widen the hidden dimensions, stack layers, and add parameters, assuming that a larger internal capacity naturally allows the model to learn more intricate, high-dimensional features.
However, a fundamental mathematical constraint often quietly undermines this scaling. A network's internal representation capacity is strictly bounded by its input layer. If the input data is mapped into the network via a low-rank or purely linear mechanism, the rest of the network—no matter how wide or deep—can become severely bottlenecked, rendering much of its parameter wealth redundant.
The Geometry of Wasted Width
To understand why this bottleneck occurs, we must look at the fundamental rule of matrix rank: the rank of a product of matrices cannot exceed the rank of its shallowest constituent.
In Transformer architectures, tokens are mapped into a continuous space via an embedding matrix before entering the self-attention layers. In a study by Wies et al., the authors demonstrated that the expressivity of a Transformer is fundamentally capped by the rank of this initial input embedding matrix.
[Input Tokens] ---> [Linear Embedding Layer] ---> [Wide Transformer Layers]
(Low Rank) (Wasted Capacity)
If the vocabulary size or the intrinsic rank of the embedding matrix is smaller than the model's hidden dimension, the network's internal states are trapped within a low-dimensional subspace. Multiplying these states by massive weight matrices downstream merely rotates or projects that same low-dimensional subspace into a larger room. As Wies et al. point out, widening a network beyond the rank of its input tokens yields sharp diminishing returns, creating substantial architectural redundancy in massive models.
This issue is not unique to language models. In Vision Transformers (ViTs), Hong et al. identified that standard linear projections of image patches create a similar representational bottleneck, limiting the model's generalization performance from the very first layer.
The Rank Collapse of Linear Projections
The bottleneck becomes even more acute in tabular foundation models, where continuous numerical features must be converted into high-dimensional vectors.
In standard architectures, a single numeric feature (such as a financial value or a physical measurement) is typically multiplied by a weight vector to project it into the model's hidden dimension. As explained by Wang et al., this linear mapping forces all possible embedding vectors for that feature to reside on a single straight line in the high-dimensional space.
Mathematically, this restricts the entire input matrix to a maximum rank of two, regardless of how wide the subsequent transformer layers are. Because a straight line cannot capture non-linear variations or complex data correlations, the high-dimensional internal states of the model remain highly redundant, wasting the capacity of large-scale architectures.
A parallel phenomenon occurs within the attention mechanism itself. Qiu et al. observed that standard Transformer blocks frequently suffer from a low-rank bottleneck when consecutive linear projections for values and outputs collapse into a single linear mapping. Without intermediate non-linearities to break the chain, the expressivity of the attention block degrades.
The Sample-Constrained Linear Regime
The rank bottleneck does not only emerge from architectural choices; it can also be strictly imposed by the volume of training data. Even if a network possesses the architectural capacity to represent complex, high-dimensional functions, it may be mathematically prevented from learning them if the sample size is insufficient.
Consider the "extensive-width" regime, where the number of training samples $n$ scales linearly with the input dimension $d$. Work by Cui et al. and Maillard et al. establishes a stark information-theoretic limit: in this data-scarce regime, the optimal learning strategy for a deep network is effectively reduced to finding a simple linear approximation of the target function.
Why does this happen? When data is limited, the complex non-linear components of a target function are statistically indistinguishable from random noise. Maillard et al. demonstrate that wide networks are trapped in this linear regression regime until the number of training samples scales quadratically ($n \propto d^2$) with the input dimension.
From an optimization perspective, this limitation is tied directly to how gradient descent updates fully connected layers. As proven by Warmuth et al., the effective rank of a fully connected network's weight matrix is fundamentally constrained by the number of training examples it encounters.
During training, gradient updates can only alter the weights along the subspaces spanned by the training inputs. If the sample size is small, the weight matrix cannot develop the rank necessary to fit sparse or orthogonal target functions, forcing the network to behave like a shallow linear model.
Breaking the Bottleneck with Early Non-Linearity
If the root of the problem is that linear input layers constrain the rank and force models into a linear regime, the solution is to introduce non-linearity before the data enters the deep architecture.
To circumvent the rank-two limitation in tabular models, Wang et al. introduced a Radial-Basis Embedding Layer (RaBEL). Instead of mapping a scalar to a straight line, this non-linear embedding maps features across a manifold, immediately expanding the rank of the input matrix and unlocking the latent capacity of the downstream transformer.
Similarly, Hong et al. designed non-linear and shared token embedding strategies for Vision Transformers to bypass the linear projection bottleneck, achieving superior downstream generalization without increasing parameter complexity. Within the network layers, Qiu et al. demonstrated that introducing gating mechanisms—which act as element-wise non-linearities—prevents consecutive linear projections from collapsing into low-rank mappings.
Ultimately, these insights reveal that network capacity is not merely a function of width or parameter count. True expressivity requires a careful alignment between the rank of the input representation, the architectural mechanisms, and the sample scale. By breaking the linear input bottleneck, we ensure that the internal width of a model is fully utilized rather than mathematically wasted.
Key Foundational Work
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021) — contributed 6 concept elements to this direction.
A case where a spindly two-layer linear network decisively outperforms any neural network with a fully connected input layer (2021) — contributed 2 concept elements to this direction.
Bayes-optimal Learning of Deep Random Networks of Extensive-width (2023) — contributed 2 concept elements to this direction.