Back to the Future - Sequential Alignment of Text Representations

Johannes Bjerva, Wouter M. Kouw, Isabelle Augenstein

Introduction

Language is inherently dynamic, evolving over time as new concepts emerge and existing words take on new meanings. In the context of Natural Language Processing (NLP), this "data drift" poses a significant challenge: models trained on historical data often see a degradation in performance when applied to contemporary text. For instance, before 2018, "Bert" and "Elmo" typically referred to characters; today, they frequently denote neural network architectures.

Semantic drift of word embeddings over time, showing how terms like 'bert' and 'ernie' shift relative to other concepts.

To address this without the prohibitive cost of continuous retraining, this paper introduces Sequential Subspace Alignment (SSA). SSA is a computationally efficient domain adaptation technique that aligns text embeddings across time steps, allowing models to generalize to future data using only a minimal amount of new labeled examples.

Methodology: Sequential Subspace Alignment

Instead of modifying the weights of a pre-trained language model like BERT, SSA operates on the extracted embeddings. The method leverages Subspace Alignment, a technique that identifies the principal components (subspaces) of source (past) and target (present) data and learns a linear transformation to align them.

Conceptual illustration of subspace alignment, where the principal components of two distributions are aligned via a linear transformation.

The authors propose several variants:

  • Unsupervised Subspace Alignment: Aligns the global structure of datasets, though it often fails when different classes drift in divergent directions.
  • Semi-Supervised Subspace Alignment (SSA): The primary contribution, which aligns data on a per-class basis. By using a tiny set of labeled target examples (2–10 per class), it employs a 1-nearest-neighbor approach to pseudo-label the remaining target data, enabling precise class-aware alignment.
  • Unbounded Time Extension: To handle long-term drift (e.g., across a decade), a binary branching tree structure iteratively merges adjacent time steps into a unified joint space.
  • Clustering-Enhanced Alignment: Incorporates k-means clustering to maintain semantic coherence in joint spaces over extended periods.

Experimental Evaluation

The effectiveness of SSA was tested across three diverse tasks: Academic Paper Acceptance (PeerRead), Named Entity Recognition (Broad Twitter Corpus), and Rumor Stance Classification (RumourEval 2019).

Key Results

SSA consistently outperformed standard baselines, such as training on all historical data or only the most recent time step.

  • PeerRead: SSA with clustering achieved 69.56% accuracy, significantly higher than the 58.82% achieved by training on unaligned historical data.
  • Twitter NER: SSA reached an F1 score of 71.35, surpassing the "all data" baseline of 67.86.
  • Rumour Stance: The semi-supervised approach yielded an F1 of 38.65, compared to 31.13 for the baseline.

Discussion and Insights

A major finding is that simply accumulating more historical data can be counterproductive; without alignment, old data acts as noise due to distribution drift. SSA proves that alignment is remarkably data-efficient, requiring very few target labels to be effective. However, the authors emphasize that global alignment is generally insufficient for NLP, as individual semantic categories often drift independently.

Limitations

SSA assumes that classes form cohesive clusters. If a class "splinters"—for example, if a single rumor topic evolves into multiple distinct sub-discussions—the linear alignment may fail. Furthermore, the method's reliance on a small number of anchor labels makes it sensitive to label noise; a single mislabeled example can misalign an entire class distribution.

Summary updated

Abstract

Language evolves over time in many ways relevant to natural language processing tasks. For example, recent occurrences of tokens 'BERT' and 'ELMO' in publications refer to neural network architectures rather than persons. This type of temporal signal is typically overlooked, but is important if one aims to deploy a machine learning model over an extended period of time. In particular, language evolution causes data drift between time-steps in sequential decision-making tasks. Examples of such tasks include prediction of paper acceptance for yearly conferences (regular intervals) or author stance prediction for rumours on Twitter (irregular intervals). Inspired by successes in computer vision, we tackle data drift by sequentially aligning learned representations. We evaluate on three challenging tasks varying in terms of time-scales, linguistic units, and domains. These tasks show our method outperforming several strong baselines, including using all available data. We argue that, due to its low computational expense, sequential alignment is a practical solution to dealing with language evolution.
BibTeX
@inproceedings{bjerva2020back,
  author = {Johannes Bjerva and Wouter M. Kouw and Isabelle Augenstein},
  title = {{Back to the Future - Sequential Alignment of Text Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6240},
}
Figure