Spam filters, content moderators and phishing detectors that run on streams of text share one asymmetry. Letting a harmful document through costs far more than flagging a harmless one. They also share a blind spot. Documents the model calls "safe" are almost never checked by a human, so when attackers change tactics, the costly misses pile up exactly where no labels arrive. Standard drift detectors watch error rates, which lag behind the change and require labels. Label-free detectors watch the whole input stream and treat every change as equally important. Neither is aimed at the place where the expensive errors hide.
We propose a label-free monitor aimed at that place. Each incoming document is summarized by its topic mixture, meaning the proportions of a few dozen latent themes that a topic model such as Latent Dirichlet Allocation assigns to it. From that mixture we take two things: the document's dominant topic and its topic entropy, which measures how spread out the mixture is. Earlier work shows that entropy separates focused human writing from scrambled or machine-assembled text, and from keyword-stuffed text that has been made artificially narrow. The monitor tracks how documents the classifier currently labels "safe" are distributed over (dominant topic, entropy) cells. It raises an alarm only when that distribution shifts in cells where a miss would be costly. When it fires, it sends documents from the drifted cells to human audit. That single action both blocks likely misses and collects labels where the model is going blind.
Core Claim
The claim is a comparison at a fixed audit budget. Concentrating audits on drifted topic-entropy cells inside the predicted-safe region should detect cost-relevant drift earlier, and accumulate lower total asymmetric cost, than three alternatives: random audits, confidence-based detectors, and embedding-distance detectors. This should hold in text streams where adversaries shift their writing style.
The mechanism rests on three findings. First, Using LDA to detect semantically incoherent documents showed that LDA topic entropy separates coherent text from synthetic corruptions. On sentence salads its equal error rate was 0.88%, against 48.8% for mixture-model entropy, and it clearly beat per-word likelihood. Second, Studying Ranking-Incentivized Web Dynamics found that authors who game rankings deliberately narrow their content, which shows up as low entropy. Together these imply that adversarial drift should move documents along the entropy axis in both directions. Third, Early Concept Drift Detection via Prediction Uncertainty proved that error rate can stay flat while the underlying distribution shifts. That result motivates watching a distribution rather than an error count.
What is new is the combination of three choices that prior work makes separately or not at all. We restrict monitoring to the region where costly errors are censored. We stratify that region by topic and entropy. And we use the detector's alarm to allocate audits that also feed adaptation.
Research Program
The base system has three parts. A frozen LDA model is fit on a reference window, with about $K=50$ topics, the setting at which the incoherence work found entropy detection performed best. A downstream binary classifier sits on top. A two-threshold routing policy completes the system, following Learning to Offload with Low Regret under Asymmetric Misclassification Costs. That paper proved that the optimal policy under asymmetric costs sends an "ambiguous" band of confidence scores to a stronger decision-maker, and it learns the band online despite censored feedback. Its base model, however, is static and binary, and it reacts to drift only by widening the band globally. It has no notion of where in content space the drift is.
We partition the topic simplex (the space of all possible topic mixtures) into cells, each defined by a dominant topic and an entropy bin. Over sliding windows we maintain the histogram of predicted-safe documents across these cells. We test the reference histogram against the current one using the adaptive-bucket chi-square approach that PUDD applies to uncertainty values. PUDD itself needs true labels to compute its uncertainty index, whereas our statistic needs none. The alarm score for cell $c$ is
$$ s_c = C_{\mathrm{FN}} \, \hat{p}_c \, \hat{D}_c , $$
where $C_{\mathrm{FN}}$ is the cost of a false negative, $\hat{p}_c$ is the current fraction of predicted-safe documents in cell $c$, and $\hat{D}_c$ is that cell's contribution to the chi-square divergence. A fixed per-window audit budget $B$ is spent across cells in proportion to $s_c$. A small floor of uniformly random audits is kept so that every cell eventually yields some labels. This floor addresses the censoring problem: without it, cells that never alarm would never be checked.
Audited labels drive local adaptation. Only the classifier's behavior on drifted cells is updated, using a low-dimensional version of the regional idea in Regional Concept Drift Detection and Density Synchronized Drift Adaptation. That work showed that text drift is regional and that flushing the whole buffer discards valid knowledge. It also noted that its k-nearest-neighbor density search in a 500-feature spam space is a bottleneck. Our cells live in a 50-topic simplex, so that cost largely disappears.
Two known weaknesses of the representation shape the design. Entropy estimates are unreliable below roughly 80–100 words, so short documents are pooled by author or thread where the data allows. Their results are also reported separately rather than hidden. The topic model is fixed, and documents dominated by vocabulary it has never seen get flattened toward high entropy. We treat this as a signal rather than a defect. We test it, and we also test a periodic refit of the topic model as an ablation.
Baselines
The naive reference is a frozen classifier with a fixed confidence threshold and the same audit budget $B$ spent uniformly at random over predicted-safe documents. Random auditing gives an unbiased estimate of the miss rate, and it is what a careful operator would do by default.
The first detector baseline is the Incremental Kolmogorov-Smirnov test (Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test) applied to the classifier's confidence on predicted-safe documents. It is fast and label-free, it is the obvious one-dimensional alternative, and its authors report matching supervised toplines while requesting only 42% to 47% of labels. When it fires, it receives the same audit budget, spent on the lowest-confidence safe documents.
The second detector baseline is the centroid-distance detector of Task-Sensitive Concept Drift Detector with Constraint Embedding. It is designed to ignore harmless input drift and react only to drift that matters for the decision boundary, which is the same goal we pursue through costs. We train it on sentence embeddings of the reference window.
For adaptation quality, the strongest reported system on the one real text stream in our file is LDD-DSDA. It reports F1 of 0.9610 on Spam Filtering (Regional Concept Drift Detection and Density Synchronized Drift Adaptation), though under full labeling. Beating it would mean matching that F1 within one point while using at most our audit budget of labels, and achieving lower cumulative asymmetric cost. We also run the two-threshold policy of HI-LCB-2D as a cost-aware routing baseline that has no content-level drift signal. No reported result was found on Lacuna for HI-LCB-2D's numerical cost on any text stream.
Two ablations isolate the key ingredients. The first monitors all documents rather than only predicted-safe ones. The second keeps the dominant-topic cells but drops the entropy axis. If neither ablation loses, the central claim is false.
All methods share the same LDA or embedding preprocessing, classifier, audit budget, window sizes, and threshold-tuning budget. Tuning is done on a held-out prefix of each stream only.
Why not just…?
Why not just audit at random? This is the real threat. With enough budget, random audits estimate the miss rate without bias and need no detector at all. Our bet is that at small budgets, perhaps 1% of traffic, drift concentrated in a few cells is found much sooner by targeted audits. The decisive comparison is detection delay and cumulative cost as a function of $B$. If random auditing matches us at realistic budgets, the targeting idea fails, and we will say so.
Why not just run a KS test on classifier confidence? Confidence is one-dimensional, and it is produced by the very model whose blind spots we are looking for. An adversary who pushes spam deep into the "safe" region raises confidence rather than lowering it, and topic entropy is built to catch exactly that kind of text: salad-like or keyword-narrowed. The incoherence results above show that thematic structure catches corruption that local likelihood misses. Whether this advantage carries over from likelihood to classifier confidence has not been shown, and the IKS-on-confidence baseline tests it directly.
Why not just use sentence-embedding distances, as LaGoNN or centroid detectors do? Like a Good Nearest Neighbor: Practical Content Moderation and Text Classification shows that frozen embeddings with nearest-neighbor labels beat fine-tuned RoBERTa by over 10 average-precision points on imbalanced moderation tasks, so embeddings are a strong representation. Distance measures do not, however, say whether new content is broadening or narrowing in theme. It is also possible that dense embeddings make topic entropy redundant. The centroid baseline settles this. As a further check, we also cluster embedding space into the same number of cells, so that the comparison isolates the representation rather than the cell structure.
Proposed Experiments
Real text streams with labeled drift points are rare. The systematic review Concept Drift Adaptation in Text Stream Mining Settings: A Systematic Review found only 2 of 48 papers with such data. We therefore use two kinds of stream.
The first is controlled. We order Reuters RCV1 by date and inject the word-salad, sentence-salad and Markovian corruptions from the incoherence study as a "harmful" class. The corruptions start in a few topics only, and we add keyword-narrowed variants that lower entropy. This gives known drift onsets in both entropy directions.
The second is the real Spam Filtering stream used by LDD-DSDA, on which we fit LDA directly over the bag-of-words counts.
Content moderation datasets from the LaGoNN suite and Jigsaw comments supply an exploratory short-text setting. They have no native timestamps, so we create drift by changing which topics the toxic examples are drawn from over time.
The metrics are delay from drift onset to detection, false alarms on purely benign drift, and cumulative asymmetric cost at cost ratios of 5, 20 and 100 to 1, all plotted against audit budget. We also report F1, which allows comparison with the 0.9610 figure.
The decisive checks are three. Targeted auditing must beat random auditing at budgets of 1% or less. Removing either the restriction to predicted-safe documents or the entropy axis must cost measurable detection delay. And the monitor must stay quiet when only benign documents drift.
References
- Using LDA to detect semantically incoherent documents
- Studying Ranking-Incentivized Web Dynamics
- Early Concept Drift Detection via Prediction Uncertainty
- Learning to Offload with Low Regret under Asymmetric Misclassification Costs
- Regional Concept Drift Detection and Density Synchronized Drift Adaptation
- Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test
- Task-Sensitive Concept Drift Detector with Constraint Embedding
- Like a Good Nearest Neighbor: Practical Content Moderation and Text Classification
- Concept Drift Adaptation in Text Stream Mining Settings: A Systematic Review