Specializing Word Embeddings (for Parsing) by Information Bottleneck

Xiang Lisa Li, Jason Eisner

Abstract

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.

Introduction

Pre-trained contextual embeddings like BERT and ELMo are incredibly powerful because they encode a wealth of syntactic and semantic information. However, this "all-in-one" nature can be a double-edged sword. For a specific task like dependency parsing—which focuses on the structural relationships between words—much of the semantic information (the "meaning" of the words) is essentially noise that can lead to overfitting.

The standard solution is fine-tuning, where the entire massive model is updated for the task. While effective, fine-tuning is computationally expensive and risks memorizing training data. This paper proposes an alternative: non-linearly compressing these embeddings to keep only the information necessary for parsing using the Variational Information Bottleneck (VIB).

Methodology: The Variational Information Bottleneck

The goal of the Information Bottleneck (IB) principle is to find a compressed representation $T$ that minimizes the mutual information with the input $X$ (compression) while maximizing the mutual information with the target $Y$ (prediction).

The VIB parsing architecture: ELMo embeddings are compressed into taggings $t$ before being fed to the dependency parser.

The authors minimize the objective function $\mathcal{L}_{IB} = -I(Y; T) + \beta I(X; T)$, where the parameter $\beta$ controls the "narrowness" of the bottleneck. They introduce two flavors of compression:

  • Discrete (VIBd): Compresses embeddings into a small set of $k$ discrete tags, similar to Part-of-Speech (POS) tags.
  • Continuous (VIBc): Compresses embeddings into low-dimensional vectors sampled from a Gaussian distribution.

A deep biaffine dependency parser acts as the "decoder," attempting to reconstruct the parse tree from these compressed variables. To ensure the learned tags remain local and interpretable, the authors add a novel term to the objective that penalizes the model if the compressed tag $T_i$ relies too much on surrounding context rather than the word's identity.

Experimental Setup

The researchers evaluated their method on the Universal Dependencies (UD) v2.3 corpus across 9 languages with diverse syntactic structures: Arabic, Chinese, English, French, Hindi, Portuguese, Russian, Spanish, and Italian. They used pre-trained ELMo embeddings as the base input.

Key Results and Insights

Syntactic Filtering

The bottleneck effectively acts as a filter. By showing that a classifier could no longer predict a word's semantic stem from the compressed vector, but could still predict its POS tag, the authors proved they were successfully "squeezing out" meaning to focus on structure.

t-SNE visualizations of embeddings at different levels of compression. As $I(X;T)$ decreases, the embeddings cluster more clearly by syntactic category (POS tags).

Performance and Generalization

  • Parsing Accuracy: The continuous VIB model (VIBc) outperformed the uncompressed ELMo baseline in 8 of the 9 languages tested.
  • Learned Tags vs. Gold Tags: In 6 out of 9 languages, the automatically learned discrete tags resulted in better parsing accuracy than using human-annotated "Gold" POS tags at the same level of granularity.
  • The "Goldilocks" Zone: Moderate compression improves performance on unseen test data. Too little compression leads to overfitting, while the right amount forces the model to ignore idiosyncratic semantic details.
  • Efficiency: The VIB approach is highly efficient, training on 10,000 sentences in just 100 seconds on a single GPU—significantly faster than full model fine-tuning.

Conclusion

By specializing word embeddings through an information bottleneck, the authors demonstrate that "less is more" for task-specific performance. While these specialized embeddings are currently task-specific, the framework offers a path toward more interpretable and efficient NLP models. Future work may extend this to unsupervised tasks, potentially learning a general generative grammar by intentionally "forgetting" specific word identities.

Summary updated

Abstract

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.
BibTeX
@inproceedings{li2020specializing,
  author = {Xiang Lisa Li and Jason Eisner},
  title = {{Specializing Word Embeddings (for Parsing) by Information Bottleneck}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=F9j6h_QG6hX},
}
Figure