TL;DR
This paper reveals that learned position embeddings in transformer models, crucial for understanding sequence order, occupy a surprisingly low-dimensional subspace. This structure is characterized by periodic, orthogonal components, suggesting a more efficient and structured way of encoding positional information than previously assumed.
What Problem Does This Paper Address?
Transformers, the backbone of modern NLP, are inherently order-agnostic. To overcome this, they rely on position embeddings, which inject information about the sequence order of tokens. While these embeddings are essential, their internal structure and dimensionality have remained largely underexplored. Existing methods often use fixed, non-learnable embeddings or learnable ones without a deep understanding of their underlying organization. This paper aims to fill this gap by quantitatively analyzing the dimensionality and structure of learned absolute position embeddings across various popular transformer models.
Methods
The core methodology employed is Principal Component Analysis (PCA). PCA is a standard dimensionality reduction technique that identifies the principal axes of variation in a dataset. By applying PCA to the learned position embeddings (and word embeddings for comparison) from several transformer models, the authors aimed to:
- Quantify dimensionality: Determine how many dimensions are truly necessary to capture the variance in position embeddings.
- Identify structural patterns: Analyze the nature of the principal components to understand how positional information is encoded.
The authors extracted embedding matrices from pre-trained models and then performed PCA on these matrices. They compared the variance explained by the top principal components for position embeddings versus word embeddings, and also visualized the principal components themselves.
Datasets
The paper does not introduce new datasets. Instead, it analyzes pre-trained transformer models. The specific models analyzed include:
- Twelve different versions of ALBERT (base-v1, base-v2, large-v1, large-v2, xlarge-v1, xlarge-v2, xxlarge-v1, xxlarge-v2)
- Three versions of BERT (base-uncased, base-cased, large-uncased, large-cased)
The embeddings were obtained from models provided by Hugging Face.
Key Insights and Ideas
- Position Embeddings are Low-Dimensional: Unlike word embeddings, which spread their variance across many dimensions, position embeddings are highly concentrated in a low-dimensional subspace, typically using less than 10% of the available dimensions. This suggests a more efficient encoding mechanism for positional information.
- Periodic and Orthogonal Structure: The dominant principal components of position embeddings exhibit a smooth, periodic oscillation as a function of position. These components often appear in pairs (like sine and cosine) and are mutually orthogonal, suggesting a structured, harmonic basis for encoding order.
- Model-Agnostic Structure: This low-dimensional, periodic structure is observed consistently across different transformer models (ALBERT and BERT variants), indicating a general principle of how positional information is learned.
- Translation Equivariance Emerges Quickly: By reconstructing position embeddings using only the top principal components, the authors demonstrate that translation-equivariant structure (where embeddings shift predictably with position) is rapidly created. This contrasts with classic sinusoidal embeddings.
- Outliers at Special Tokens: The first and last position embeddings (often corresponding to special tokens like
[CLS]and[SEP]) show outlier behavior in 2D PCA plots, likely due to their fixed roles in training.
Key Results
- Dimensionality Reduction: For ALBERT and BERT models, 50% of the variance in position embeddings is explained by the first 1.4-5.5% of principal components. In contrast, word embeddings require 19-30% of PCs to achieve the same.
- Number of Components: Position embeddings effectively lie on a subspace of 10 to 20 dimensions, whereas word embeddings utilize the full dimensionality of the embedding space.
- Rotational Structure: PCA visualizations reveal a clear rotational structure in the embedding space, where position embeddings complete full clockwise turns as position indices increase.
Limitations and Future Work
Limitations:
- Linear Subspace Assumption: PCA only considers linear subspaces. The true dimensionality of the position embedding manifold might be lower if non-linear relationships exist.
- Focus on Absolute Embeddings: The study is limited to learned absolute position embeddings and does not analyze alternative schemes like Rotary Position Embeddings (RoPE).
- Input Embeddings Only: The analysis is confined to input embeddings. The structure and dimensionality of these embeddings might change as they pass through subsequent layers of the transformer.
- Self-Attention Interaction: The study does not specifically analyze how these low-dimensional manifolds influence the transformer's self-attention mechanism.
Future Work:
- Streamlining Embeddings: The findings suggest opportunities to streamline embeddings and reduce computational demands by using factorized parameterizations with different dimensionalities for position and word embeddings.
- Initializing Embeddings: Learned position embeddings could be initialized or incentivized to have rotational structure, potentially using sines and cosines with learnable frequencies.
- Exploring Other Domains: Investigating position embeddings in other domains, such as computer vision, is suggested.
- Understanding Self-Attention: Further research into how these low-dimensional manifolds interact with self-attention could shed light on why this structure emerges.
- Re-using Embeddings: The consistent structure across models suggests that re-using learned position embeddings from older models might be a useful general strategy.
Summary updated