Keeping Label-Name-Only Classifiers Current as Language Drifts

Some text classifiers are built from nothing but the names of the categories. A system is told "sports, politics, arts & culture," expands each name into a list of related words using a pretrained language model, labels unlabeled documents by matching against those lists, and then trains on its own guesses. These label-name-only classifiers can match supervised models trained on dozens of hand-labeled documents per class. All of them assume that the word lists stay valid. On social media they do not. "Mask" moves from fashion to public health, new shows and players become the main signals for "arts" or "sports," and old signals disappear. We propose to rebuild each category's word list at every time period from unlabeled text alone. A language model matched to that period proposes new indicative words. Old words are retired when their meaning in context has drifted, which we detect by comparing how a word is represented in old and new tweets. The classifier is then retrained on the refreshed lists, so it stays current without any new human labels.

The case for doing this rests on two findings in the literature that have not been connected. Temporal-split evaluations show that supervised topic classifiers degrade sharply as tweets age (Twitter Topic Classification). The best unsupervised detector of meaning change, plain cosine similarity of contextual embeddings, beats fine-tuned detectors (TempoWiC). If drift mostly damages the word lists, then a cheap and label-free refresh of those lists should recover most of the lost accuracy. The proposal is built so that we learn this even if the answer is no.

Core Claim

Our hypothesis is that under temporal drift, label-name-only classifiers fail first in the step that builds category vocabularies, and only second in the classifier itself. If that is right, a refresh that adds period-matched words and retires semantically drifted ones should close most of the gap between a stale weakly supervised classifier and one rebuilt for the current period. A second hypothesis is that warm-starting self-training from the previous period's model locks in outdated signals. Self-training means retraining on the model's own confident predictions. When it starts from an old model, confident guesses based on old vocabulary can crowd out the new vocabulary, so a refresh step is needed rather than continued self-training alone.

Relation to Prior Work

Three methods supply the machinery. WeSTClass generates pseudo-documents from seed words and self-trains. LOTClass builds category vocabularies by asking BERT's masked-word predictor which words can stand in for the label name, then self-trains. PIEClass keeps only pseudo-labels on which several classifier views agree, which limits error amplification. We reuse LOTClass's vocabulary construction and PIEClass's agreement filter. All three assume a single static corpus, and we drop that assumption.

Three temporal-adaptation papers motivate the gap and also constrain the design. VIBE separates past-only, future-only and time-invariant topics using a small amount of unlabeled future data, but it needs fully labeled source-period data. Our setting has no document labels in any period. Non-parametric temporal adaptation keeps the encoder frozen and appends labeled tweets to a nearest-neighbor store. It needs a labeled stream, which we do not assume, and its authors warn that the frozen encoder eventually ages. Chen et al. find emerging terms through frequency bursts and send the bursty tweets to human annotators. They concede that the method cannot see meaning change that happens without a frequency spike. Our retirement step targets exactly that blind spot, and it labels nothing by hand.

Two resources supply the signals we use. TimeLMs provide quarterly language-model checkpoints, which we use as period-matched masked-word predictors. They also document roughly a 10% performance loss after one year. TempoWiC supplies the unsupervised similarity test for meaning shift, with a reported best macro-F1 of 70.33. That figure reminds us the detector is imperfect, so retirement decisions must be soft.

Research Program

At each period $t$ we have the fixed label names and a pool of unlabeled tweets from that period. The procedure has three steps.

The first step is to add words. For each category, we take current-period tweets in which the category's existing vocabulary words occur and mask those words. The TimeLM checkpoint for period $t$ then proposes substitutes. Words that are proposed across many contexts for the same category join that category's list, following the LOTClass recipe but drawing on period-$t$ contexts.

The second step is to retire words. For each vocabulary word $w$, we compute $s(w) = \cos(\bar h_{t-1}(w), \bar h_t(w))$. Here $\bar h_t(w)$ is the average contextual embedding of $w$ over period-$t$ tweets, computed under one fixed reference encoder so that the two periods are comparable. Each word's weight in the list is scaled down as $s(w)$ falls. We use soft down-weighting rather than hard deletion because the detector errs often.

The third step is to retrain. LOTClass-style masked category prediction and self-training run on period-$t$ data using the refreshed lists, with PIEClass's agreement filter applied to pseudo-labels. We compare three ways of initializing the classifier: starting fresh each period, warm-starting from period $t-1$, and warm-starting with a refreshed vocabulary.

Alongside the method, we will measure entrenchment directly. For warm-started self-training, we will track what fraction of accepted pseudo-labels are triggered only by words that are past-exclusive, meaning their frequency fell between periods. We will also track how accuracy on emerging subtopics changes across self-training rounds.

Baselines

The naive reference is the stale model. LOTClass and PIEClass are built once on period-0 unlabeled data with the 2019 TimeLM and applied unchanged to later test periods. Its gap to the supervised ceiling defines how much loss there is to recover.

The strongest weakly supervised competitor is the rerun-from-scratch baseline. PIEClass is rebuilt independently at each period on that period's unlabeled data using the current TimeLM checkpoint. This is the obvious thing a practitioner would do, because these methods need no labels to rerun. It gets the same unlabeled data, the same checkpoints, the same number of self-training rounds, and the same tuning budget as our method, all on a validation slice from period 0 only. Our method beats it only if it improves macro-F1 on later periods or matches that F1 at clearly lower compute and with less period-to-period instability in predictions.

The strongest zero-shot competitor is Qwen3-Reranker-8B, which BTZSC reports at 0.72 average macro-F1 across 22 datasets. No result on TweetTopic temporal splits was found on Lacuna, so we will run it ourselves. It uses the same label names plus, as a variant, our refreshed vocabularies inserted into its label descriptions.

The supervised ceiling comes from Twitter Topic Classification. On the single-label temporal split, TimeLM-19 reports 70.0% F1 and 86.4% accuracy. On the multi-label split, the best model, TimeLM-21, reports 58.8% F1.

Two ablations isolate the key ingredients. The first adds words without retiring any. The second retires words without adding any. A further ablation replaces similarity-based retirement with Chen et al.'s burstiness score, which tests whether meaning-aware retirement matters beyond frequency.

Why not just…?

Why not just rerun PIEClass from scratch each period? This is the most serious threat. Rerunning already rebuilds the vocabulary from current text and a current checkpoint. Our method improves on it only if carrying the vocabulary across periods with explicit retirement adds something. Two things could provide that. First, the carried-over vocabulary keeps stable words and stops each period's rerun from wandering toward whatever is incidentally frequent in that period. Second, a refresh costs only a vocabulary pass rather than a full rebuild. The rerun baseline and the add-only ablation settle this. If rerunning matches the full method in F1, the contribution reduces to the entrenchment measurement and the cost comparison, and we will report it that way.

Why not just prompt a large zero-shot model? SuperTweetEval finds that prompted generative models trail TimeLM-Large, which is fine-tuned on tweets, on most tweet tasks. Rerankers are stronger, but they were pretrained after 2021, so they have already seen the "drifted" TweetTopic test period. Their strength on that split may not carry over to genuinely new text. We will report reranker results with this caveat. We will also test the reranker on a held-out pool of the newest tweets we can collect, scored by a small expert-labeled set. If the reranker stays strong there, drift is not the bottleneck for zero-shot topic classification, and that is a finding we would report.

Why not just continue pretraining and keep the old vocabulary? Our method already uses continually pretrained TimeLM checkpoints. The question is whether a current model paired with a stale word list is enough. The ablation that uses the period-$t$ checkpoint with the period-0 vocabulary answers this directly.

Why not label 50–100 trending tweets? Chen et al. show that this works for named-entity tagging. We include it as an upper reference that requires human labels, with 100 burst-selected tweets labeled at each period. The test is how much of that labeled gain the label-free refresh recovers.

Proposed Experiments

The main testbed is TweetTopic's single-label six-class temporal split, with training-period tweets (2019–2020) used only as unlabeled data and the 2020–2021 test set divided into quarters. Multi-label TweetTopic is the harder secondary test, because broad labels such as "Arts & Culture" are known to fail in label-name methods, and this category also decayed most in the supervised study. Unlabeled pools per quarter come from TweetTopic's own collection period and, for later weeks, from the 2021 longitudinal hashtag tweets used by the non-parametric study. Metrics are macro-F1 per quarter, the slope of decay over quarters, and per-class F1 on Arts & Culture. We also report the entrenchment fraction described above, and agreement between our retirement scores and TempoWiC's gold labels on the words the two share.

The decisive check has three outcomes. If the full refresh beats the stale model by a margin similar to the gap between random-split and temporal-split supervised training, and beats rerun-from-scratch on later quarters, the vocabulary hypothesis holds. If refresh and rerun tie, warm-starting is unnecessary, but we expect the entrenchment analysis to still explain why warm-started self-training alone fails. If neither recovers the loss while the reranker does, the bottleneck lies outside vocabulary construction, and the field should move toward verbalizer-based zero-shot models for drifting text.