Text classifiers get worse as language moves on. A Twitter model trained on 2020 data loses close to 10% after one year of newer data (TimeLMs). Practitioners usually respond with one of two fixes: keep pre-training on fresh unlabeled text, or pay annotators to label fresh examples. This proposal argues that the choice between them can be predicted ahead of time from how the label was produced. Some labels are produced by the author: a star rating, or the sentiment the writer meant to express. In these cases the label is a cause of the text. Other labels are assigned afterwards by someone who reads the text, such as an annotator tagging entities or a fact-checker issuing a verdict. In these cases the text is a cause of the label. The principle of independent causal mechanisms says the two parts of a causal system change independently: how often causes occur, and the rule that turns a cause into an effect. If that holds, the direction of labeling tells us which part of the joint distribution time is likely to change. That in turn tells us which cheap repair should work.
The concrete prediction concerns labels the author produced. For these, much of the decay over time should come from a shift in how often each label occurs. That shift can be corrected using only unlabeled future text. For labels an annotator assigned after reading, decay should come mainly from new kinds of input the model has not seen. Re-estimating label frequencies should then do almost nothing, while fresh pre-training or reweighting of inputs should help. We test this on identical texts carrying both kinds of label, and we build a cheap test that tells which regime a new task is in. The goal is a principled answer to the deployment question "should my next dollar go to unlabeled text or to labels?"
Core Claim
Write $X$ for the text and $Y$ for the label at time $t$. If the label causes the text, the natural pieces are $P_t(Y)$, how often each label occurs, and $P_t(X\mid Y)$, how each label is expressed. If the text causes the label, the pieces are $P_t(X)$ and $P_t(Y\mid X)$, the reading rule. The independent change principle formalized in CD-NOD (Causal Discovery from Heterogeneous/Nonstationary Data) says that changes over time in the two causal pieces are statistically independent. Changes in the two pieces of the reverse factorization, by contrast, are coupled.
Our claim is that this asymmetry shows up as a measurable difference in which repairs work. The quantity is the recovery ratio of a repair method $m$:
$$R_m(s\to t) = \frac{A_m(s\to t) - A_{\text{static}}(s\to t)}{A_{\text{oracle}}(t) - A_{\text{static}}(s\to t)}.$$
Here $A_{\text{static}}(s\to t)$ is the accuracy at time $t$ of a model trained on labeled data from time $s$. $A_m$ is the accuracy after applying repair $m$. $A_{\text{oracle}}(t)$ is the accuracy of the same architecture trained on labeled data from time $t$. $R_m=1$ means the repair closes the whole gap.
The hypothesis has two parts. For author-produced labels, prior-only re-estimation should give substantially positive recovery and should beat input reweighting. For reader-assigned labels on the same texts, the ordering should flip. When neither repair closes the gap, the residual should track a direct measurement of mechanism drift: either $P_t(X\mid Y)$ or $P_t(Y\mid X)$, depending on direction.
This is a different question from the closest work. Causal Direction of Data Collection Matters is a meta-analysis of published results. It found that semi-supervised learning helps anticausal tasks (+1.70 mean) far more than causal ones (+0.04), and that domain adaptation helps causal tasks (+5.18) more than anticausal ones (+1.26). It compared static, unordered domains, and it compared different tasks, so task identity and direction are confounded. We ask the question over ordered time, with direction varied while the text is held fixed. Those two published findings also point in opposite directions for temporal drift. The semi-supervised result suggests unlabeled future text should help author-labeled tasks. The domain-adaptation result suggests input-side adaptation should help reader-labeled tasks. Settling which effect dominates over time is part of the contribution.
Research Program
The first component is a paired-label corpus, which removes the confound in the meta-analysis. We start from timestamped review data already used for drift research: the Amazon and Yelp chronological splits in On the Impact of Temporal Concept Drift on Model Explanations, and the Amazon reviews in Characterizing and Measuring Linguistic Dataset Drift. Every review already has an author-produced star rating, which is the label-causes-text case. For a stratified sample of a few thousand reviews per year, we collect annotator sentiment judgments made from the text alone. That gives a reader-assigned label on exactly the same inputs.
This design fixes the reader's rule by construction, because all annotators are present-day people. A reviewer may object that this builds in the invariance we want to find. It does, deliberately, for the reader-labeled arm. The question is whether a model trained on it still decays, and whether that decay behaves like input coverage loss rather than label-frequency change. Two further datasets supply naturally occurring reader-assigned labels where the rule itself may drift. One is the 2014–2019 Twitter named-entity data of Mitigating Temporal-Drift, annotated year by year. The other is the fact-checking splits in the explanation benchmark. On these the prediction becomes a real risk.
The second component is a common harness. Each model is a RoBERTa-base classifier trained on an early time window and tested on every later window. It is paired with one of five repairs, each under the same unlabeled-text and compute budget. The repairs are: nothing; prior-only re-estimation, in which an EM procedure re-fits label frequencies from the model's predictions on unlabeled future text and adjusts the output layer; reweighting of training examples toward the future input distribution, using a domain classifier; continual masked-language pre-training on future unlabeled text in the style of TimeLMs; and self-training on future unlabeled text. We record $R_m$ for every pair of training and test windows, label type, and repair.
The third component is a temporal independent change test for text. It handles new tasks whose labeling process is undocumented, and it measures mechanism drift directly. Each time slice is summarized in a low-dimensional way. Input drift is the vocabulary, part-of-speech-structure and contextual-semantic drift scores of Characterizing and Measuring Linguistic Dataset Drift. Label-frequency drift is the estimated change in $P_t(Y)$. Conditional drift in each direction is estimated from a small labeled probe of a few hundred examples per slice. For text-causes-label, it is the change in a slice-specific classifier's predictions on a fixed reference set. For label-causes-text, it is the change in per-class drift scores.
Following CD-NOD, we compute a normalized Hilbert–Schmidt independence statistic across slices. This statistic is a kernel measure of whether two sequences of changes are dependent. We compute it once for the pair (change in $P(X)$, change in $P(Y\mid X)$) and once for the pair (change in $P(Y)$, change in $P(X\mid Y)$). The direction with weaker dependence is the inferred one. The residual drift in the inferred conditional predicts what unlabeled repairs cannot fix. The final output is a decision rule: repair label frequencies when the task looks author-labeled, adapt inputs when it looks reader-labeled, and buy labels in proportion to the measured conditional drift.
The program has a known limitation. Both the eigen-analysis artifact detector (Detecting non-causal artifacts in multivariate linear regression models) and CD-NOD were developed for low-dimensional or continuous signals. We sidestep the high-dimensional problem by testing on drift summaries rather than raw embeddings. A further risk is that with roughly 20 quarterly slices the independence test will have little statistical power. We therefore also use monthly slices where volume permits, and we report bootstrap intervals.
Baselines
The naive reference point is the static model with no repair, scored at every future window. Its decay curve sets the denominator of $R_m$, and it lets us check that our data reproduce the roughly 10%-in-one-year decay reported by TimeLMs.
The strongest unlabeled baseline is continual pre-training on fresh text, as in TimeLMs, which pre-trained on 4.2 million tweets per quarter and reduced decay on TweetEval. The file gives no exact recovered-accuracy figure for this, only the degradation figure and the qualitative improvement. We run it with the same unlabeled token budget as every other unlabeled repair. Beating it means that, on author-labeled data, prior-only re-estimation reaches equal or higher recovery at a small fraction of the compute.
The strongest label-buying baseline is trend-based annotation selection from Mitigating Temporal-Drift. It scores n-grams by their frequency rise between periods and labels the tweets that contain them, improving named-entity F1 by 2.70% to 5.71% over uniform temporal sampling. We give it the same annotation budget that our decision rule spends on probes plus targeted labels. Beating it means higher accuracy at equal labels on the named-entity data, or equal accuracy with fewer labels.
The strongest selection baseline is the oracle-free practical one: label a small validation set from the future window and pick whichever repair scores best on it. We give it exactly the same number of labeled future examples as our probe. Beating it means that the direction-based rule, or the independence test, chooses the right repair more often when labels per slice are very few.
Two ablations isolate the key ingredient. The first replaces the inferred direction with a random one, and also with "always the best repair on average", to show the direction itself carries information. The second runs the decision rule on the drift summaries alone, without the independence test. This shows whether the causal structure adds anything beyond the decomposed drift metrics of Characterizing and Measuring Linguistic Dataset Drift. Those metrics reduced the error of out-of-domain accuracy prediction by 16.8% relative to holistic metrics, and that is the bar for the degradation-prediction part of our work.
Why not just…?
Why not just always continually pre-train on new text? It is the default in practice, and TimeLMs shows it helps. But it is reactive, it costs constant compute, and the evidence from Mitigating Temporal-Drift suggests that domain-specific pre-training alone did not prevent named-entity decay, which is a reader-labeled task. Our harness tests this directly. If continual pre-training yields equally high recovery for both label types on the paired corpus, then direction does not matter for this repair, and our central claim fails.
Why not just label the trending examples? Trend selection works for bursty topics, but it only detects frequency spikes. It cannot see a word gaining a new meaning without becoming more frequent, which is the gradual drift documented in Cultural Shift or Linguistic Drift?. It also cannot see a shift in label frequencies, which costs no new labels to fix. If label-frequency change dominates decay on author-labeled reviews, trend selection spends annotation money on a problem that could be fixed for free. The matched-budget comparison will show whether this happens.
Why not just pick the repair using a small labeled future validation set? This is the real threat. With enough future labels it should match any principled rule. Our rule is only worth having when labels are scarce, or when the direction of a new task must be decided before any repair has been tried. We therefore sweep the probe size from a few dozen to several hundred examples per slice. We report the label budget at which validation-based selection catches up. If it catches up at a few dozen labels, the practical contribution shrinks to the scientific finding about what drifts.
Why not just predict decay with decomposed drift metrics? Those metrics already beat holistic embedding distances at predicting accuracy drops. However, they describe how much the input changed, not which factor of the joint distribution changed. The second ablation measures whether adding the direction-aware conditional drift term lowers the prediction error further.
Proposed Experiments
The first decisive check runs on the paired-label review corpus. For each pair of training and test windows, we compare $R_m$ for prior-only re-estimation against input reweighting, separately for star ratings and annotator labels on the same reviews. The claim survives only if the ordering of the two repairs flips between label types, with a confidence interval excluding zero. If prior-only re-estimation helps both label types equally, then label-frequency shift is simply common in reviews and has nothing to do with causal direction.
The second check uses the naturally reader-labeled named-entity and fact-checking data, where the reading rule itself may drift. We test whether the leftover gap after the best unlabeled repair correlates with the measured drift in $P_t(Y\mid X)$ across slices, more strongly than with the input drift scores alone.
The third check is blind direction inference on tasks with unclear labeling processes, such as AGNews topics and the TweetEval tasks evaluated in TimeLMs. We compare the independence test's verdict against a documented labeling procedure where one exists. We then score the end-to-end decision rule against the matched-budget baselines above.
A final, cautionary probe concerns historical part-of-speech tagging. On the Penn historical corpora, unsupervised feature adaptation cut error by 30% on a reader-labeled task (Part-of-Speech Tagging for Historical English), showing that input-side adaptation can be very effective there. The same work warns that annotation standards also shift across eras. We will use this corpus to see whether our test flags that shift as drift in the reading rule rather than misreading it as input change.
References
- TimeLMs: Diachronic Language Models from Twitter
- Causal Discovery from Heterogeneous/Nonstationary Data.
- Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
- On the Impact of Temporal Concept Drift on Model Explanations
- Characterizing and Measuring Linguistic Dataset Drift
- Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp
- Detecting non-causal artifacts in multivariate linear regression models
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
- Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change
- Part-of-Speech Tagging for Historical English