Xiang Lisa Li, Jason Eisner Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Introduction
Pre-trained contextual embeddings like BERT and ELMo have revolutionized NLP by capturing a broad spectrum of linguistic information. However, for specialized tasks like dependency parsing—identifying grammatical relationships between words—this "bit of everything" approach can introduce irrelevant noise, potentially leading to overfitting. This paper explores how to extract the "syntactic essence" of a word while discarding unnecessary semantic or contextual details using the Information Bottleneck principle.
Methodology: The Variational Information Bottleneck
The authors propose using the Variational Information Bottleneck (VIB) to "specialize" general-purpose embeddings. The goal is to find a compressed representation $T$ of an input embedding $X$ that remains highly predictive of the target parse tree $Y$.
The VIB objective balances two competing goals:
- Maximizing $I(Y; T)$: Retaining sufficient information to accurately predict the grammatical structure.
- Minimizing $I(X; T)$: Forgetting as much of the original, high-dimensional input as possible to reduce noise.
Architecture
The model employs a Bi-LSTM "Token Encoder" to generate the bottleneck variable $T$ from ELMo embeddings. This is followed by a deep biaffine dependency parser acting as the "Decoder." A key innovation is the Type Encoder, which ensures that the learned tags $T$ behave like stable "word types" rather than just encoding the entire sentence context.
The researchers implemented two versions of this bottleneck:
- Discrete (VIBd): Compressing embeddings into a finite set of learned "tags" (e.g., 64 or 128 categories).
- Continuous (VIBc): Compressing embeddings into very low-dimensional vectors (e.g., as small as 5 dimensions).
Key Insights and Results
The method was evaluated on the Universal Dependencies (UD) 2.3 dataset across nine diverse languages, including Arabic, Chinese, English, and Russian.
The Compression-Performance Trade-off
The study revealed a characteristic saturation curve: as the amount of information retained ($I(X; T)$) increases, parsing accuracy (LAS) and mutual information with the tree ($I(Y; T)$) improve until they reach a plateau. Interestingly, moderate compression often helps generalization; keeping too much information can lead to overfitting on the training data.
Efficiency and Sufficiency
One of the most striking findings was that continuous tags with only 5 dimensions performed nearly as well as 256-dimensional vectors. This suggests that the syntactic information required for parsing is remarkably low-dimensional. Furthermore, the automatically learned discrete tags (VIBd) consistently outperformed gold-standard Part-of-Speech (POS) tags for the parsing task, providing a more useful signal to the parser than traditional human-defined categories.
Conclusion
By applying the Information Bottleneck principle, the authors demonstrate that "less is more" in the context of task-specific embeddings. Specializing embeddings through VIB not only improves parsing accuracy across multiple languages but also results in significantly more efficient models. These compressed representations are faster to process on CPUs, making high-quality parsing more accessible for resource-constrained applications.
Summary updated