Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment

Piotr Bojanowski, Onur Celebi, Tomás Mikolov, Edouard Grave, Armand JoulinarXiv 2019 · OpenReview

Abstract

In this paper, we focus on the problem of adapting word vector-based models to new textual data. Given a model pre-trained on large reference data, how can we adapt it to a smaller piece of data with a slightly different language distribution? We frame the adaptation problem as a monolingual word vector alignment problem, and simply average models after alignment. We align vectors using the RCSLS criterion. Our formulation results in a simple and efficient algorithm that allows adapting general-purpose models to changing word distributions. In our evaluation, we consider applications to word embedding and text classification models. We show that the proposed approach yields good performance in all setups and outperforms a baseline consisting in fine-tuning the model on new data.

Introduction

Pre-trained word embeddings like word2vec or fastText are foundational to modern NLP. However, they face two significant challenges: specialization and temporal drift. General-purpose models trained on web crawls often lack the nuance required for niche domains like medicine or law. Furthermore, language is dynamic; new terms emerge, and the semantics of existing words evolve over time.

The conventional solution, fine-tuning, often leads to "catastrophic forgetting," where the model loses general statistical knowledge while attempting to learn new distributions. Retraining from scratch is frequently unfeasible due to the sheer volume of original data. Researchers at Facebook AI Research (FAIR) have introduced a "Monolingual Alignment" method that addresses these issues by framing domain adaptation as a vector alignment problem.

The Monolingual Alignment Method

The proposed approach, "align-and-average," follows a three-step workflow:

  1. Local Model Training: A new, compact model ($Y$) is trained on the specific target dataset ($S_1$).
  2. Alignment: A linear mapping ($Q$)—specifically an orthogonal transformation—is identified to rotate the new vectors into alignment with the pre-trained vectors ($X$). This alignment utilizes the RCSLS (Relaxed Cross-Domain Similarity Local Scaling) loss function to mitigate the "hubness" problem.
  3. Averaging: Once aligned, the vectors are averaged. For words present in both datasets, the new vector is the average of the original and the transformed new vector. For novel words, the transformed new vector is used directly.

This logic extends to text classifiers by aligning word vector spaces and then averaging classifier weights. The orthogonal nature of the transformation ensures that the underlying scoring functions remain mathematically compatible.

Experimental Results

Vocabulary Expansion

The researchers evaluated the method on a word analogy task designed to simulate the appearance of new words. By intentionally removing words from a large reference set ($S_0$) and retaining them in a smaller set ($S_1$), they tested the model's ability to integrate new vocabulary.

Table 1: Performance on vocabulary expansion tasks, comparing the proposed RCSLS+Fine method against baselines.

As shown in Figure 1, the proposed RCSLS+Fine method achieved 72.1% accuracy on "In-vocab" questions, notably outperforming standard fine-tuning (66.8%) and even surpassing the "Oracle" model trained on the union of all data (71.2%).

Temporal Adaptation

To test adaptation to language change over time, the researchers used the Yelp 2019 Challenge dataset, comparing reviews from 2013–2014 against those from 2018.

Table 2: Accuracy on Yelp temporal adaptation across different sizes of the new dataset ($S_1$).

The results in Figure 2 demonstrate that when the new dataset ($S_1$) is small (e.g., 10k samples), the alignment method significantly outperforms standard fine-tuning (75.1% vs. 72.9%). This highlights the method's efficiency in low-data regimes.

Model Merging

The approach also serves as an effective alternative to ensembling. In experiments on Sogou News and Amazon Reviews, the alignment method matched the performance of voting ensembles while producing a single, compact model, avoiding the computational overhead of running multiple separate models.

Conclusion

Monolingual alignment offers a robust, computationally efficient way to update pre-trained models. By preserving the global structure of the vector space through orthogonal transformations, it avoids the pitfalls of standard fine-tuning. While the method depends on sufficient vocabulary overlap for reliable alignment, it represents a significant step toward continuous model adaptation and lexicon extension in dynamic linguistic environments.

Summary updated

Abstract

In this paper, we focus on the problem of adapting word vector-based models to new textual data. Given a model pre-trained on large reference data, how can we adapt it to a smaller piece of data with a slightly different language distribution? We frame the adaptation problem as a monolingual word vector alignment problem, and simply average models after alignment. We align vectors using the RCSLS criterion. Our formulation results in a simple and efficient algorithm that allows adapting general-purpose models to changing word distributions. In our evaluation, we consider applications to word embedding and text classification models. We show that the proposed approach yields good performance in all setups and outperforms a baseline consisting in fine-tuning the model on new data.
BibTeX
@misc{bojanowski2019updating,
  author = {Piotr Bojanowski and Onur Celebi and Tomás Mikolov and Edouard Grave and Armand Joulin},
  title = {{Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment}},
  year = {2019},
  eprint = {1910.06241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.06241},
}
Figure