TL;DR
This paper argues that successful transformer language models implicitly learn translation-invariant positional encodings. They propose a new, efficient, and interpretable mechanism called Translation-Invariant Self-Attention (TISA) that can improve performance and reduce positional parameters, even potentially replacing traditional position embeddings.
What Problem Does This Paper Address?
Transformer models, the backbone of modern NLP, are inherently permutation-invariant. To understand sequential data like text, they need a way to encode the order of words. Traditional methods use "position embeddings," which are learned vectors added to word embeddings. However, these methods have drawbacks: they can be parameter-heavy, struggle with sequences longer than those seen during training, and their learned behavior isn't always clear. The authors aim to understand how transformers actually learn positional information and propose a more principled, efficient, and interpretable alternative.
Methods
The core of the paper involves two main parts:
- Analysis of Learned Positional Embeddings: The authors analyze the inner product of position embeddings ($P = E_p E_p^T$) in various pre-trained transformer models. They hypothesize that if positional information is translation-invariant, this matrix should exhibit a "Toeplitz" structure, meaning diagonals are constant. They quantify this using the coefficient of determination ($R^2$). They also analyze how self-attention scores ($Q K^T$) are affected by position embeddings.
- Proposed Translation-Invariant Self-Attention (TISA): Based on their analysis, they propose TISA. This mechanism parameterizes positional information directly within the self-attention mechanism, rather than relying solely on separate position embeddings. TISA uses a "positional scoring function" ($f_\theta(k)$) that depends only on the relative distance ($k = j-i$) between tokens. This function is modeled as a sum of radial basis functions, allowing for flexible and interpretable positional biases. TISA can either supplement existing position embeddings or replace them entirely.
Datasets
The paper uses standard GLUE tasks for evaluation. Specific datasets mentioned include:
- SST-2 (Stanford Sentiment Treebank)
- MNLI (Multi-Genre Natural Language Inference)
- QQP (Quora Question Pairs)
- STS-B (Semantic Textual Similarity Benchmark)
- COLA (Corpus of Linguistic Acceptability)
- MRPC (Microsoft Research Paraphrase Corpus)
- QNLI (Question-answering NLI)
- RTE (Recognizing Textual Entailment)
The models are evaluated using pre-trained ALBERT base v2 models.
Key Insights and Ideas
- Emergent Translation Invariance: The authors provide strong evidence that position embeddings in successful transformer models are not arbitrary but learn to be translation-invariant. This means the effect of a token's position depends more on its distance from other tokens than its absolute position in the sequence.
- Toeplitz Structure: The inner product matrix of position embeddings ($E_p E_p^T$) tends to become more "Toeplitz" (diagonals are constant) as models train and perform better. This is a strong indicator of translation invariance.
- Interpretability of TISA: Unlike traditional position embeddings, TISA's positional scoring functions are directly interpretable. The authors visualize these functions, showing how different attention heads learn different positional biases (e.g., attending to the next token, previous token, or a wider context).
- Parameter Efficiency: TISA requires significantly fewer trainable parameters than traditional position embeddings, especially for large models, making it a more efficient approach.
- Potential for Replacing Embeddings: The results suggest that TISA can be used effectively even without traditional position embeddings, offering a more streamlined architecture.
Key Results
- Performance Improvement: When TISA is added to ALBERT base v2 models, it leads to relative error reductions of 0.4% to 6.5% on GLUE tasks, with an average median F1 score increase of 0.4 points.
- Parameter Reduction: TISA adds a minuscule number of parameters (e.g., 2,160 for a specific configuration) compared to the millions of parameters in standard position embeddings.
- Qualitative Analysis: Visualizations of TISA's scoring functions reveal interpretable patterns, showing how attention heads specialize in different relative positional relationships. This provides a window into the model's positional reasoning.
- Training Speed: The pretraining initialization scheme for TISA converges very quickly (under 20 seconds on a CPU).
Limitations and Future Work
- Pretraining Strategy: The experiments primarily focus on adding TISA to already pre-trained models. The authors acknowledge that they did not have the resources to conduct full pre-training of TISA-only models from scratch, which might yield even better results.
- Inductive Bias for Arbitrary Length: While TISA is designed to be inductive and handle arbitrary lengths, the paper doesn't present experiments specifically demonstrating this capability in detail, beyond stating its potential.
- Future Work: The authors suggest that TISA's inductive and localized nature could enable faster pre-training on shorter sequences, with later scaling to longer sequences. They also propose TISA's applicability beyond NLP, to domains like time series, speech, and computer vision.
Summary updated