Introduction
Language is inherently dynamic, evolving over time as new concepts emerge and existing words take on new meanings. In the context of Natural Language Processing (NLP), this "data drift" poses a significant challenge: models trained on historical data often see a degradation in performance when applied to contemporary text. For instance, before 2018, "Bert" and "Elmo" typically referred to characters; today, they frequently denote neural network architectures.
To address this without the prohibitive cost of continuous retraining, this paper introduces Sequential Subspace Alignment (SSA). SSA is a computationally efficient domain adaptation technique that aligns text embeddings across time steps, allowing models to generalize to future data using only a minimal amount of new labeled examples.
Methodology: Sequential Subspace Alignment
Instead of modifying the weights of a pre-trained language model like BERT, SSA operates on the extracted embeddings. The method leverages Subspace Alignment, a technique that identifies the principal components (subspaces) of source (past) and target (present) data and learns a linear transformation to align them.
The authors propose several variants:
- Unsupervised Subspace Alignment: Aligns the global structure of datasets, though it often fails when different classes drift in divergent directions.
- Semi-Supervised Subspace Alignment (SSA): The primary contribution, which aligns data on a per-class basis. By using a tiny set of labeled target examples (2–10 per class), it employs a 1-nearest-neighbor approach to pseudo-label the remaining target data, enabling precise class-aware alignment.
- Unbounded Time Extension: To handle long-term drift (e.g., across a decade), a binary branching tree structure iteratively merges adjacent time steps into a unified joint space.
- Clustering-Enhanced Alignment: Incorporates k-means clustering to maintain semantic coherence in joint spaces over extended periods.
Experimental Evaluation
The effectiveness of SSA was tested across three diverse tasks: Academic Paper Acceptance (PeerRead), Named Entity Recognition (Broad Twitter Corpus), and Rumor Stance Classification (RumourEval 2019).
Key Results
SSA consistently outperformed standard baselines, such as training on all historical data or only the most recent time step.
- PeerRead: SSA with clustering achieved 69.56% accuracy, significantly higher than the 58.82% achieved by training on unaligned historical data.
- Twitter NER: SSA reached an F1 score of 71.35, surpassing the "all data" baseline of 67.86.
- Rumour Stance: The semi-supervised approach yielded an F1 of 38.65, compared to 31.13 for the baseline.
Discussion and Insights
A major finding is that simply accumulating more historical data can be counterproductive; without alignment, old data acts as noise due to distribution drift. SSA proves that alignment is remarkably data-efficient, requiring very few target labels to be effective. However, the authors emphasize that global alignment is generally insufficient for NLP, as individual semantic categories often drift independently.
Limitations
SSA assumes that classes form cohesive clusters. If a class "splinters"—for example, if a single rumor topic evolves into multiple distinct sub-discussions—the linear alignment may fail. Furthermore, the method's reliance on a small number of anchor labels makes it sensitive to label noise; a single mislabeled example can misalign an entire class distribution.
Summary updated