TransDrift: Modeling Word-Embedding Drift using Transformer

Nishtha Madaan, Prateek Chaudhury, Nishant Kumar, Srikanta Bedathur

TL;DR

This paper introduces TransDrift, a novel Transformer-based model that predicts how word embeddings change over time due to evolving language use (data drift). By accurately modeling this drift, TransDrift significantly improves the performance of downstream NLP tasks, even when minimal or no new data is available from the drifted distribution.

What Problem Does This Paper Address?

Word embeddings, like those generated by Word2Vec, are fundamental to many Natural Language Processing (NLP) applications. They represent words as dense vectors, capturing semantic relationships. These embeddings are often shared across various tasks. However, language is dynamic; word meanings and usage patterns shift over time. This "data drift" means that embeddings trained on older data may become outdated and less effective for current text. For instance, the word "vacation" might be associated with "beach" in summer but "skiing" in winter. If an NLP model relies on static embeddings, its performance can degrade as the underlying data distribution changes. A key challenge is that when drift occurs, there might be very little or no new data available from the drifted distribution to retrain embeddings, making it difficult to adapt.

Methods

The core of TransDrift is a Transformer architecture. The model takes the word embeddings from a previous time-step ($E_t$) as input and predicts the embeddings for the next time-step ($E_{t+1}$). The Transformer's self-attention mechanism allows it to consider the relationships between all words' embeddings simultaneously, enabling it to learn complex, non-linear drift dynamics.

During training, TransDrift is fed embeddings from two time-steps, $D_t$ and $D_{t+1}$. It learns to map $E_t$ to $E_{t+1}$. A key innovation is that TransDrift can also optionally incorporate a small amount of data ($D_{small}$) from the drifted distribution. This $D_{small}$ is used to train a small set of embeddings ($E_{small}$), which are then provided as additional context to the Transformer, further improving prediction accuracy. The model is trained using a cosine embedding loss.

Datasets

The authors evaluated TransDrift on three datasets:

  • Synthetic Dataset: A custom-generated dataset designed to simulate drift.
  • Yelp Academic Dataset: Used for multi-label star classification.
  • Amazon Customer Review Dataset: Used for sentiment analysis tasks, specifically on electronics, kitchen, and DVD reviews.

For each dataset, they created drift instances by splitting data into two time-steps ($D_1$ and $D_2$) and a small subset of $D_2$ ($D_{small}$).

Key Insights and Ideas

  • Language is a moving target: The paper strongly emphasizes that word semantics are not static and that ignoring this drift leads to performance degradation in NLP tasks.
  • Transformers are good at capturing drift dynamics: The Transformer's ability to model complex interactions between word embeddings is crucial for accurately predicting how they will change. Simple additive models are insufficient.
  • Adaptation without data is possible (and effective): TransDrift demonstrates that it's possible to predict embedding drift even with zero data from the new distribution, which is a significant advantage in real-world scenarios where data scarcity is common.
  • Small data can help a lot: Even a tiny amount of data from the drifted distribution can significantly boost the accuracy of TransDrift's predictions.

Key Results

  • Superior Embedding Prediction: TransDrift significantly outperforms baseline methods (No-Drift and Additive-Drift) in predicting word embeddings, achieving higher cosine similarity with ground truth embeddings across all tested datasets. For example, on the Synthetic dataset, TransDrift achieved a cosine similarity of 0.7724, compared to 0.32 for No-Drift and 0.33 for Additive-Drift.
  • Improved Downstream Task Performance: When the predicted embeddings from TransDrift were used as input for downstream tasks (sentiment analysis and star classification), they led to substantial accuracy improvements. For instance, on Amazon Review and Yelp datasets, TransDrift-based models achieved higher accuracy (e.g., 69.6% for AR-Electro vs. 60.5% for No-Drift) compared to baselines.
  • Qualitative Evidence: Visualizations of nearest neighbors for words like "place" and "happy" showed that TransDrift's predicted embeddings had more neighbors in common with the target embeddings than those from baseline models, indicating more accurate semantic capture.
  • Robustness to Data Scarcity: Even with 0% of data from the drifted distribution ($D_{small}$), TransDrift's prediction accuracy surpassed baselines. Increasing the percentage of $D_{small}$ further improved performance.

Limitations and Future Work

  • Limitations: The paper acknowledges that while TransDrift can leverage small amounts of data, its performance is still best when some data is available. The "No-Drift" baseline, while showing poor performance, represents a common practical approach, highlighting the need for drift modeling.
  • Future Work: The authors suggest exploring multi-step word embedding prediction, which would involve predicting embeddings across more than just two consecutive time-steps. This could lead to more robust models for tracking long-term semantic evolution.

Summary updated

Abstract

In modern NLP applications, word embeddings are a crucial backbone that can be readily shared across a number of tasks. However, as the text distributions change and word semantics evolve over time, the downstream applications using the embeddings can suffer if the word representations do not conform to the data drift. Thus, maintaining word embeddings to be consistent with the underlying data distribution is a key problem. In this work, we tackle this problem and propose TransDrift\footnoteCodebase: https://github.com/data-iitd/transdrift, a transformer-based prediction model for word embeddings. Leveraging the flexibility of transformer, our model accurately learns the dynamics of the embedding drift and predicts the future embedding. In experiments, we compare with existing methods and show that our model makes significantly more accurate predictions of the word embedding than the baselines. Crucially, by applying the predicted embeddings as a backbone for downstream classification tasks, we show that our embeddings lead to superior performance compared to the previous methods.
BibTeX
@inproceedings{madaan2022transdrift,
  author = {Nishtha Madaan and Prateek Chaudhury and Nishant Kumar and Srikanta Bedathur},
  title = {{TransDrift: Modeling Word-Embedding Drift using Transformer}},
  booktitle = {The Web Conference},
  year = {2022},
  url = {https://openreview.net/forum?id=dcMw0zmene},
}
Figure