Pretraining Corpus as the Source of Ensemble Diversity for Text Anomaly Detection

Detecting anomalous text, such as an off-topic document in a biomedical pipeline, a mislabeled abstract, or a scrambled record, usually works by embedding each document with a pre-trained language model and then running a classical outlier detector on the embeddings. Each embedding model has blind spots, and the blind spots follow from what the model read during pre-training. A general web-trained model has already seen most kinds of foreign text, so it tends to treat that text as familiar. A model pre-trained only on one domain separates fine distinctions inside that domain well but can be loose about text that is far from anything it has seen. We propose to treat these two failure patterns as complementary and combine them on purpose. The plan has three parts. First, build a small set ("zoo") of sibling encoders that are identical except for their pre-training corpus. Second, score every test document under each encoder with the same simple detector. Third, for each document, let a statistical test decide which encoders' verdicts to trust.

The claim, stated so that it can fail: at a matched number of ensemble members and matched compute, diversity in pre-training corpus reduces text anomaly detection errors more than diversity in random seed or architecture. The gain should be largest on the "near-distribution" shifts where current text detectors fail. If corpus diversity buys nothing over seed diversity, the idea is wrong. That outcome would itself be informative, because it would mean domain pre-training does not change where detectors fail.

Where this sits in prior work

Two lines of evidence motivate the proposal.

The first is about how much the pre-training corpus shapes representations. MonoCoder trains only on high-performance-computing code and beats much larger general code models. APEX pre-trains a 10.5M-parameter model on wireless telemetry and matches general time-series models 15–20 times its size. A Comparative Study of PEFT Methods for Python Code Generation finds that domain pre-training predicts downstream performance better than the choice of fine-tuning method. All three measure task accuracy. None of them asks how the corpus changes what looks surprising to the model.

The second line is the clearest statement of the tension we exploit. Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text shows that broadly pre-trained models "leak": they give overconfident, low-perplexity scores to foreign text they saw during pre-training. Models trained only on in-domain text instead form loose boundaries around the normal data. That paper resolves the tension by distilling one teacher into one student. We resolve it by keeping both kinds of model and selecting between them for each sample.

Three works supply the method.

  • NLP-ADBench shows that "two-step" pipelines, meaning a frozen embedding followed by a classical detector such as LOF, iForest or LUNAR, beat end-to-end text anomaly models by up to 50% AUROC. It also finds that no single detector wins on every dataset. Its embeddings are all general-purpose (BERT, OpenAI).
  • Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks shows that nearest-neighbour distance (KNN) and ViM, a method that measures how far a feature vector falls outside the main directions of the training data, beat logit-based scores. It also shows that every method fails on subtle background shifts and on in-domain text with shuffled word order.
  • ZODE supplies the combination rule. Each detector's score is turned into a p-value calibrated on in-distribution data. The Benjamini-Hochberg procedure, a standard multiple-testing correction that controls the false discovery rate, then picks which detectors are "active" for each test sample.

ZODE's zoo varies architecture and training objective on fixed image data. Our zoo holds architecture and detector fixed and varies only the pre-training corpus. This is a different, and in text so far untested, source of diversity.

TADPOLE is the nearest work joining the two directions, and it constrains our design in two ways. It uses an Isolation Forest over frozen BERT embeddings to select pre-training data, which is anomaly detection used for pre-training, whereas we use pre-training to build detectors. Its "finetuning trap" result, that frozen pre-trained features detect better than fine-tuned ones, is why we use frozen encoders throughout and never fine-tune on labels.

ADBench is a warning. Unsupervised detectors lose a median 16.43% when anomalies are duplicated or clustered, so we test the clustered-anomaly setting explicitly rather than assume it away.

Research Program

Building the zoo. We start from one public encoder, BERT-base, and run continued masked-language-model pre-training for a fixed, modest budget on each of four corpora: biomedical abstracts, computer-science papers, news, and reviews. These are the four domains in TADPOLE's evaluation. A fifth sibling continues on a general web sample drawn from FineWeb for the same number of steps. The untouched base model is a sixth member.

The control zoo has the same size and the same total compute. It consists of five continued-pre-training runs on the general web sample that differ only in random seed and data order, plus the base model. This control is what isolates the corpus effect. A secondary, larger-scale check uses the FeynTune Llama-3.1-8B variants, which were each pre-trained on a different physics subfield (hep-th, hep-ph, hep-lat). They form a ready-made corpus-diverse zoo at 8B scale, if the weights are available to us.

Scoring. Each encoder gives two scores per document. The first is KNN distance in the frozen, mean-pooled embedding space, chosen as the strongest single detector in the classical text benchmark. The second is the encoder's own pseudo-perplexity: mask each token in turn and average how poorly the model predicts it. Scores are turned into conformal p-values against a held-out set of normal documents. Benjamini-Hochberg selection runs per sample, as in ZODE. Holding the detector fixed means any gain comes from the encoders, not from detector variety.

Analysis. Before running the ensemble, we measure how correlated the encoders' per-sample scores are on normal versus anomalous data. Our hypothesis predicts a crossover by shift type. Domain-continued encoders should win on near shifts, such as a held-out category within the same domain. General encoders should win on far shifts and on disrupted text, such as shuffled word order. Score correlation between corpus-diverse siblings should also be lower than between seed siblings. If the crossover does not appear, the ensemble has no reason to help, and we would report that before tuning anything.

Baselines

The naive reference point is a single general BERT-base embedding with Isolation Forest, essentially the TADPOLE-style detector. Next to it we run the logit-free maximum-softmax-style alternative of scoring by base-model pseudo-perplexity alone. Both use the same normal-data calibration split and are expected to sit at the bottom.

The strongest existing two-step method is LUNAR over OpenAI text-embedding-3-large embeddings. It is the best configuration in NLP-ADBench, which reports 0.9732 AUROC on BBCNews and the top AUROC on 5 of 8 datasets. We run it unchanged on every benchmark below using the NLP-ADBench code. Its embedding model is far larger than ours, so beating it with a zoo of BERT-base siblings would be a strong result, while matching it at a fraction of the inference cost would still be useful.

On near-OOD news, KNN on fine-tuned RoBERTa is the strong post-hoc baseline. It reaches 99.8% AUROC on Twitter near-OOD in the classical text OOD benchmark. On intent OOD, multi-level distillation reports 92.51 AUROC on CLINC150 (Multi-Level KD), and we compare against it there. For the domain datasets we construct from CHEMPROT, RCT20K, ACL-ARC and SCIERC, no reported anomaly-detection result was found on Lacuna, so all numbers there come from our own reruns.

The key ablations hold everything fixed except one ingredient:

  • seed-diverse zoo versus corpus-diverse zoo, which tests the central claim;
  • the best single member chosen in hindsight versus per-sample selection, which tests whether selection adds anything beyond picking the right domain model;
  • uniform score averaging (the AOM and MOA rules in PyOD) versus Benjamini-Hochberg selection.

Every method gets the same normal-data split for fitting and calibration and the same hyperparameter search budget. For KNN that means $k$ chosen from a fixed grid on a normal-only validation split. Beating a baseline means a higher mean AUROC and a lower FPR at 95% true-positive rate (FPR95), each by more than the seed-to-seed standard deviation across five runs.

Why not just…?

Why not just use the matching domain encoder alone? If the test domain is known, one domain-continued encoder with KNN may capture most of the gain. This is a real threat, and the "best single member" ablation settles it directly. Our hypothesis predicts the domain encoder alone will regress on far and disrupted shifts. That prediction rests on the loose-boundary finding in Multi-Level KD. If the domain encoder alone matches the zoo on every shift type, we would report that domain continued pre-training is sufficient and drop the ensemble.

Why not just concatenate all embeddings and run one KNN? Concatenation is cheaper to deploy than per-model calibration. However, the distance in the joined space is dominated by whichever encoder has the largest spread, and it cannot express "trust model A on this sample and model B on that one." We include concatenation with per-block normalization as a baseline. If it matches per-sample selection, the selection step is unnecessary.

Why not just average the scores? Uniform averaging dilutes a single confident detector. ZODE shows that per-sample selection beats averaging on images, reducing FPR from 11.07% to 3.83% on CIFAR-10 (ZODE), but this has not been shown for text. The averaging-versus-selection ablation tests whether the result carries over.

Why not just use a larger general embedding model? NLP-ADBench shows OpenAI embeddings are already strong. But a model trained on more general data should leak more, not less, which is exactly the failure that domain siblings target. The OpenAI+LUNAR comparison on the domain-constructed datasets and on shuffled text settles this. If it wins there too, corpus diversity is outweighed by scale.

Proposed Experiments

The main suite has three parts.

  • Domain anomaly sets. We build four sets following the NLP-ADBench protocol, where one class is downsampled to under 10% to serve as the anomalies. The sources are CHEMPROT and RCT20K (biomedical) and ACL-ARC and SCIERC (computer science), the domain datasets in TADPOLE's evaluation.
  • Existing benchmarks. We add the eight NLP-ADBench datasets and the full split structure of the classical text OOD benchmark: near, far, distinct, and the shuffled-word "News Category Random" set, plus the 20 Newsgroups semantic and background shifts.
  • Clustered anomalies. We replicate ADBench's anomaly-duplication protocol by duplicating anomalies six times. ZODE-style selection can fail here if every encoder learns the anomaly cluster as normal.

Metrics are AUROC, AUC-PR and FPR95, reported per shift type. We also report the empirical false-positive rate against the nominal level implied by the conformal calibration. Inference cost is reported as forward passes per document, because the zoo multiplies cost by the number of members. We test a cheaper variant that calls only the general model plus the encoder whose domain is nearest to the target domain.

The decisive checks are three.

  1. On near-shift and shuffled-text splits, the corpus-diverse zoo must beat the seed-diverse zoo at matched size and compute.
  2. The predicted crossover must appear: domain siblings win near shifts and general models win far or disrupted shifts. Without it, any ensemble gain is ordinary variance reduction.
  3. The selection rule must keep the empirical false-positive rate near its nominal level on the domain-constructed sets.

If all three hold, pre-training corpus becomes a design choice for anomaly detectors in its own right, not only for task accuracy. If the first fails, we will have shown that domain pre-training changes what models know without changing which inputs look surprising to them.