Abstract
Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.
Introduction
Pre-trained contextual embeddings like BERT and ELMo are incredibly powerful because they encode a wealth of syntactic and semantic information. However, this "all-in-one" nature can be a double-edged sword. For a specific task like dependency parsing—which focuses on the structural relationships between words—much of the semantic information (the "meaning" of the words) is essentially noise that can lead to overfitting.
The standard solution is fine-tuning, where the entire massive model is updated for the task. While effective, fine-tuning is computationally expensive and risks memorizing training data. This paper proposes an alternative: non-linearly compressing these embeddings to keep only the information necessary for parsing using the Variational Information Bottleneck (VIB).
Methodology: The Variational Information Bottleneck
The goal of the Information Bottleneck (IB) principle is to find a compressed representation $T$ that minimizes the mutual information with the input $X$ (compression) while maximizing the mutual information with the target $Y$ (prediction).
The authors minimize the objective function $\mathcal{L}_{IB} = -I(Y; T) + \beta I(X; T)$, where the parameter $\beta$ controls the "narrowness" of the bottleneck. They introduce two flavors of compression:
- Discrete (VIBd): Compresses embeddings into a small set of $k$ discrete tags, similar to Part-of-Speech (POS) tags.
- Continuous (VIBc): Compresses embeddings into low-dimensional vectors sampled from a Gaussian distribution.
A deep biaffine dependency parser acts as the "decoder," attempting to reconstruct the parse tree from these compressed variables. To ensure the learned tags remain local and interpretable, the authors add a novel term to the objective that penalizes the model if the compressed tag $T_i$ relies too much on surrounding context rather than the word's identity.
Experimental Setup
The researchers evaluated their method on the Universal Dependencies (UD) v2.3 corpus across 9 languages with diverse syntactic structures: Arabic, Chinese, English, French, Hindi, Portuguese, Russian, Spanish, and Italian. They used pre-trained ELMo embeddings as the base input.
Key Results and Insights
Syntactic Filtering
The bottleneck effectively acts as a filter. By showing that a classifier could no longer predict a word's semantic stem from the compressed vector, but could still predict its POS tag, the authors proved they were successfully "squeezing out" meaning to focus on structure.
Performance and Generalization
- Parsing Accuracy: The continuous VIB model (VIBc) outperformed the uncompressed ELMo baseline in 8 of the 9 languages tested.
- Learned Tags vs. Gold Tags: In 6 out of 9 languages, the automatically learned discrete tags resulted in better parsing accuracy than using human-annotated "Gold" POS tags at the same level of granularity.
- The "Goldilocks" Zone: Moderate compression improves performance on unseen test data. Too little compression leads to overfitting, while the right amount forces the model to ignore idiosyncratic semantic details.
- Efficiency: The VIB approach is highly efficient, training on 10,000 sentences in just 100 seconds on a single GPU—significantly faster than full model fine-tuning.
Conclusion
By specializing word embeddings through an information bottleneck, the authors demonstrate that "less is more" for task-specific performance. While these specialized embeddings are currently task-specific, the framework offers a path toward more interpretable and efficient NLP models. Future work may extend this to unsupervised tasks, potentially learning a general generative grammar by intentionally "forgetting" specific word identities.
Summary updated