Learning to Adapt Language Models from Unlabeled Recent Text

Text classifiers go stale. A model trained on 2015 tweets can lose up to 40 F1 points on 2020 tweets. The obvious fix, continuing to pretrain the model on raw text from the new year, barely helps. The deployed system, however, already has something prior temporal methods ignore: a steady stream of unlabeled text from the present. Every week brings thousands of new documents. We propose to train a small network that reads a batch of such recent, unlabeled text and outputs a correction to the classifier. The network is trained on the model's own history. At each past moment we pretend the model was frozen, show the network the unlabeled text that came next, and score the corrected model against the labels we now know for that later period. The goal is to learn which signals in new text actually move a task's decision boundary, rather than hoping a generic language-modeling objective finds them.

This combines two research threads. One is episodic meta-learning for domain generalization, which trains a model across simulated "train here, test there" splits so it learns to cope with shift. The other is the study of temporal linguistic drift. Existing temporal domain generalization methods forecast where the model's weights or feature statistics will move next and never look at the target period. They also assume the drift is smooth. Language drift is often abrupt, so we expect our approach to help most where forecasting breaks: sudden lexical turnover, new named entities, and inverted label proportions.

Core Claim

Continued pretraining on target-era text fails because its training signal, predicting masked words, is not tied to the task. Time Waits for No One! measures temporal degradation across eight tasks and finds that this domain-adaptive pretraining gives "only minor gains," while even a little labeled target-year data helps a lot. Our claim is that a feed-forward adaptation network, trained on chronologically ordered episodes, can recover a substantial share of the labeled-data gain from unlabeled text alone. It should do so most clearly under abrupt drift, where trajectory-forecasting methods are expected to fail.

The mechanism follows Adaptive Risk Minimization. ARM meta-trains a context network that summarizes an unlabeled test batch so the predictor can use domain context. It was tested only on images and cell imaging (FEMNIST, RxRx1), and it assumes test batches come from a single homogeneous domain. That assumption is a known weakness for categorical domain generalization. For temporal deployment it holds almost by construction, because one week of tweets is a natural homogeneous batch. We change ARM in two ways. Episodes are ordered in time rather than drawn from unordered domains. The context summary is mapped into a low-rank weight update, a LoRA increment (a small product of two thin matrices added to frozen weights), on a frozen language-model backbone.

Let $\theta_s$ be the parameters of a task model fine-tuned on labeled data up to time $s$, and let $U_{t}$ be an unlabeled batch from a later window $t > s$. Let $D_{t}$ be the labeled data of that window. Let $g_\phi$ be the adaptation network that maps a batch to a low-rank weight increment, and let $\ell$ be the task loss. Meta-training solves

$$\min_\phi \; \mathbb{E}_{s < t}\; \ell\big(\theta_s + g_\phi(U_{t});\, D_{t}\big),$$

where the expectation runs over past pairs of a frozen origin and a later window. Adaptation is a single forward pass with no inner gradient loop. This avoids the second-order cost that makes MAML-style training, as in MASF, hard to scale to large models.

Relation to Prior Work

Six temporal domain generalization methods motivate and constrain the design. Four forecast the model itself:

  • DRAIN predicts future network weights with a recurrent model.
  • Koodos models weight trajectories with a differential equation linearized in a Koopman latent space (a learned coordinate system where the dynamics become linear).
  • Prompting-based TDG extrapolates per-period prompts with a small transformer.
  • MaT-LoRA fits a time-varying core inside a shared LoRA subspace and is the only one evaluated on a large language model.

Two forecast the data instead:

  • EvoS forecasts feature means and variances and standardizes with them.
  • CODA forecasts feature correlation matrices to simulate future data.

All six extrapolate from history without seeing the target period, and several explicitly assume smooth dynamics. The Koodos authors note that their method cannot handle sudden shifts. Our method does the opposite. It never extrapolates; it conditions on observed present text. We borrow MaT-LoRA's idea of a shared low-rank subspace so the generated update stays small.

Two lines of temporal NLP work set the application regime. Mitigating Temporal-Drift shows that tweets containing trending n-grams are dense in new entities, and uses them to choose what humans should label. That is evidence that the unlabeled stream carries task-relevant signal, though their method still needs annotators. Dynamic Benchmarking of MLMs shows that knowledge of updated facts peaks at the training timestamp and decays immediately after, which motivates adapting continuously rather than retraining periodically.

SPUME supplies a technique. It builds support and query sets that deliberately disagree on spurious attributes. We reuse that idea to create synthetic abrupt-shift episodes.

Research Program

The backbone is RoBERTa-base for comparability with the benchmark papers. A 1B-parameter decoder is used for the AIC comparison with MaT-LoRA. The backbone is frozen, and a LoRA adapter plus task head are trained on labeled history. The adaptation network $g_\phi$ has three parts:

  • It encodes each document in a batch of 64 to 512 unlabeled documents with the frozen backbone.
  • It pools these encodings with a small set-attention module, so the summary does not depend on document order.
  • It emits a rank-8 increment in a fixed LoRA basis, plus an additive shift to the output logits.

The logit shift lets the network express label-prior change, which Time Waits for No One! identifies as a driver of the worst degradation: the Republican-to-Democratic tweet ratio flips between 2015 and 2020.

The main difficulty is that real history supplies few transitions. A task with six annual eras gives at most fifteen ordered pairs. We address this in three ways:

  • Finer windows. We cut history into monthly or quarterly windows where timestamps allow.
  • Shared meta-training across tasks. One $g_\phi$ is meta-trained across all tasks in the Luu suite, with a task embedding as input.
  • Synthetic abrupt episodes, in the spirit of SPUME. We resample a later window's label proportions away from the origin's. We also construct query windows whose most frequent entities and trending n-grams never appear in the origin period, using the trend score of Mitigating Temporal-Drift.

At deployment we freeze $g_\phi$ and run it once per incoming window.

Baselines

Every method gets the same backbone, the same labeled history up to cutoff $T$, the same hyperparameter search budget of 20 configurations, and the same GPU-hour cap. Every method that can use unlabeled data is given the identical unlabeled target window.

Naive references. Offline pooled ERM trains once on all history. LastDomain fine-tunes on the latest period only. IncFinetune fine-tunes sequentially through the periods. These follow the baseline definitions in MaT-LoRA. Beating them is necessary but not interesting.

Forecasting baseline. MaT-LoRA is the strongest existing temporal method for language models. On AIC, the file reports only that MaT-LoRA variants "achieve accuracies in the low 90s, outperforming baselines"; it gives no exact figure. We will rerun it under our protocol on AIC, the news publisher task, and Yelp. Winning means higher accuracy at forecast horizons of one to three years on the high-drift tasks, with no loss on Yelp. For the Huffpost and Arxiv benchmarks we also run EvoS. The file gives no exact EvoS figures on Arxiv: there is no reported result on Lacuna for EvoS accuracy on Arxiv or Huffpost.

Unlabeled-data baselines. These see the same target text we do:

  • Domain-adaptive pretraining on the target window, followed by the original head.
  • Test-time re-estimation of normalization statistics, the language-model analogue of ARM-BN.
  • Label-prior re-estimation by expectation-maximization on the target window, with no change to features.

Beating these is the central test of the claim.

Upper bound. Fine-tuning with 100 labeled target-window examples. We report the fraction of this gain, measured over LastDomain, that each unlabeled method recovers.

Ablations isolating the key ingredients. Three variants each remove one ingredient:

  • Meta-train $g_\phi$ on randomly ordered, unordered episodes instead of chronological ones.
  • Restrict $g_\phi$ to the logit shift only, with no weight increment.
  • Remove the synthetic abrupt episodes.

Why not just…?

Why not just continue pretraining on target-era text? Luu et al. (Time Waits for No One!) report only minor gains from this across eight tasks, far below what minimal labeled data gives. That result is the gap we target. We run domain-adaptive pretraining on exactly the same unlabeled window, so if it matches us, the claim fails.

Why not just forecast the drift with MaT-LoRA or Koodos? These methods should do well when drift is smooth, and on stable tasks they may tie with us. Their own authors state the limitation: they never see the target period and assume smoothness. Our decisive split compares smooth stretches against windows with label flips or entity turnover. If forecasting matches us on the abrupt windows too, the case for conditioning on the present text collapses.

Why not just correct the label prior? This is the most serious threat. The worst reported degradation, political affiliation on Twitter with a temporal degradation score of 7.72, is partly a label-proportion flip, which classical prior re-estimation handles without any meta-learning. We take this seriously in two ways. Expectation-maximization prior correction is a full baseline, and the logit-shift-only ablation measures how much of our gain is label shift. If those two account for nearly all of it on the Luu classification tasks, we will reframe the contribution around the token-level NER task. There the prior over labels moves less and new entities dominate.

Why not just label a few trending tweets? The active selection of Mitigating Temporal-Drift gains 2.70% to 5.71% F1 over the baseline on Twitter NER, but it requires annotators every cycle. We treat it as a costed alternative. We report how many labeled trending tweets are needed to match the unlabeled adaptation, which tells a practitioner what our method saves.

Proposed Experiments

Classification. The main testbed is the Luu temporal misalignment suite: political affiliation, the AI conference venue task (ICML vs. AAAI), publisher classification, media frames, and Yelp ratings. We train up to year $T$ and evaluate on $T{+}1$ through $T{+}k$. We report per-horizon F1 and the temporal degradation score, the average performance loss per year of train/test gap. Political affiliation, the most volatile task (score 7.72), is the stress case. Yelp (score 0.26) is the do-no-harm control: adaptation must not hurt a stable task.

Token-level labeling. We use the Temporal Twitter NER dataset, training on 2014–2018 and testing on 2019, and adapt per monthly window in the test year. Longer horizons and many classes. We use Huffpost and Arxiv, comparing against EvoS and continual-learning methods.

Three checks decide the outcome:

  1. Recovered labeled gain. Our method must recover a larger fraction of the 100-label fine-tuning gain than domain-adaptive pretraining, test-time normalization, and prior re-estimation. It must do this on at least the three high-drift classification tasks and on NER.
  2. Abruptness. We group test windows by how much their label proportions and entity vocabulary changed relative to $T$. Our advantage over MaT-LoRA should grow with abruptness. If it shrinks or stays flat, the core claim fails.
  3. Chronology. The chronologically trained $g_\phi$ must beat the same network trained on unordered episodes. Otherwise ordering, the claimed link between the two research directions, adds nothing beyond ordinary ARM-style meta-learning.

We also test robustness to batch size, from 16 to 512 documents, and to windows deliberately mixed across years. This probes the batch-homogeneity weakness of ARM in the setting where we argue it matters least.