Abstract
Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.
Introduction
Contextual word embeddings such as ELMo and BERT have revolutionized natural language processing by capturing a broad spectrum of linguistic features, ranging from low-level syntax to high-level semantic and world knowledge. However, for a specific downstream task like dependency parsing—which involves identifying the grammatical structure of a sentence—much of this captured information acts as "noise."
While fine-tuning is the standard adaptation technique for these models, it is computationally intensive and risks overfitting to the training data. This research addresses a fundamental question: Can we nonlinearly compress these embeddings to isolate the specific syntactic "signal" required for parsing? By employing the Information Bottleneck (IB) principle, we aim to create representations that are both efficient and specialized, stripping away irrelevant context while retaining the core properties—such as part-of-speech, tense, or transitivity—necessary to build an accurate parse tree.
Methodology
The core of our approach is the Variational Information Bottleneck (VIB). The goal is to find a compressed representation $T$ of the input word embedding $X$ that retains maximum information about the target output $Y$ (the parse tree).
The Objective Function
We minimize a loss function that balances two competing objectives:
- Compression: Reducing the mutual information $I(X; T)$ between the original embedding and the new representation. This forces the model to discard irrelevant information.
- Prediction: Maximizing the mutual information $I(Y; T)$ between the new representation and the parse tree. This ensures that the "syntactic core" is preserved.
Implementation and the Type Encoder
We use variational approximations to create a tractable neural network objective. We explore two flavors of compressed representations:
- Continuous (VIBc): The embedding is mapped to a low-dimensional Gaussian vector.
- Discrete (VIBd): The embedding is mapped to one of $k$ latent categories, effectively inducing a task-specific Part-of-Speech tag set.
A critical component of our method is the Type Encoder. To prevent the model from "cheating" by relying too heavily on neighboring words to identify a tag, we add a penalty term. This forces the compressed tag to rely more on the specific word's identity (its "type") rather than just its surroundings, ensuring the learned tags are truly representative of the word's inherent syntactic role.
Experiments and Results
We evaluated our approach using the Universal Dependencies (UD) treebanks across 9 diverse languages: Arabic, Hindi, English, French, Spanish, Portuguese, Russian, Italian, and Chinese. This variety ensures the method's robustness across different word orders and morphological systems.
Performance and Efficiency
The continuous VIB representation (VIBc) outperformed uncompressed ELMo and standard dimensionality reduction (MLP) in 8 out of 9 languages. Surprisingly, compressing embeddings to as few as 5 dimensions resulted in parsing performance nearly identical to high-dimensional versions. This suggests that the "syntactic core" of a word is actually quite small. Furthermore, the method is exceptionally fast, training on 10,000 sentences in about 100 seconds per epoch on a single GPU.
Interpretability and Syntactic Specialization
Our discrete tags discovered by the model often outperformed traditional "Gold" Part-of-Speech (POS) tags for the purpose of parsing. This implies that standard linguistic categories (like "Noun" or "Verb") might not be the most efficient categories for a machine to use when building a tree.
Visualization via t-SNE demonstrates that as compression increases (moving from panel a to c), word clusters become cleaner and more aligned with syntactic functions. The model successfully "forgets" non-syntactic information; for example, it merges common and proper nouns into a single cluster because the distinction is functionally irrelevant for building the dependency tree.
Conclusion
By applying the Information Bottleneck principle, we can effectively "distill" general-purpose embeddings into task-specific representations. This process not only improves parsing accuracy and computational efficiency but also provides a window into the minimal information required for syntactic analysis. Future work could extend this specialization approach to other semantic or syntactic tasks, or apply it to the latest generation of Large Language Models to better understand their internal representations.
Summary updated