Scholarly corpora are used to build citation graphs, compute impact metrics, and pretrain or prompt language models. All of these uses quietly assume that every indexed record is a research paper. That assumption is wrong. Digital libraries also index journal mastheads, conference front matter, committee rosters, memorial tributes, and tutorial abstract volumes, and they often file them under the same research_article tag as real papers. Nobody can currently say how large this contamination is, how it varies across fields and years, or whether it moves any number people care about.
We propose to treat this as a measurement problem rather than a classification contest. The method comes from a neighbouring field. Public-health and urban-mobility researchers validate cheap, noisy, high-volume signals against slower authoritative counts, such as Twitter health messages against laboratory surveillance, or Strava rides against city intersection counts. We apply the same logic to scholarly indexes. A cheap document-type classifier runs on the whole corpus. A small, stratified human audit, together with external registries of accepted papers, corrects the classifier's aggregate counts. The deliverable is a set of per-field, per-venue, per-year contamination estimates with honest confidence intervals, plus a direct measurement of how much that contamination distorts bibliometrics and metadata extraction.
Core Claim
Downstream users of a scholarly corpus mostly need to know how much non-research material sits in a given slice and whether it changes their results. Knowing whether one particular record is a masthead matters less. Answering the first question by counting classifier predictions is biased whenever the classifier's errors differ across slices. They plausibly will. Mastheads and front matter have distinctive titles and entity-dense text. Tributes and tutorial syntheses read like ordinary scholarly prose. A field heavy in tributes will therefore be undercounted, and a field heavy in mastheads will be counted accurately.
Our claim is twofold. First, a cheap classifier combined with a stratified audit gives prevalence estimates that are unbiased by construction and much tighter than the audit alone. Second, these estimates match independent registry counts at the venue-year level more closely than raw classifier counts do.
Where the Pieces Come From
The validation lens is borrowed from cross-dataset validation of digital traces. Participatory Epidemiology correlated crowdsourced phone reports with CDC influenza figures at r = 0.74. Imrit et al. matched Strava counts to Toronto intersection counts at R² = 0.69. VisualCit chained automatic filters with crowd checks and reached r = 0.91 against survey ground truth. These papers motivate the design: aggregate alignment with an authoritative count is the right test for a high-volume noisy signal.
They also constrain it. The same literature warns that the traces are skewed by who produces them. Imrit et al. note that Strava users are a fitness-oriented subpopulation, and #FoodPorn found that what people post diverges from what they do. A single correlation coefficient can therefore hide subgroup bias. That warning is exactly why we stratify the audit rather than validating one global number.
Three lines of scholarly-corpus work supply the setting and the gap. S2ORC carefully validates citation linking and parsing across 81.1M records but has no document-type validation step. Content-Informed Index reweights citations using abstract embeddings, validated only on curated corpora of genuine papers. MOLE extracts 30+ metadata fields with LLMs and assumes the input is a paper presenting a dataset. None of the three asks how many of its inputs are not papers.
The classifier recipe comes from FineWeb. FineWeb distilled an LLM judge into an embedding-plus-linear classifier to make quality scoring tractable at web scale. We reuse that distillation step. Its punctuation heuristics we do not expect to transfer, because tributes are fully punctuated prose.
The evidence that the problem is real is concrete. Mastheads such as IEEE Transactions on Cybernetics publication information and VLDB Front Matter carry research_article descriptors in OpenAlex and DBLP. The COLING tutorial abstracts volume is tagged conference_paper.
Research Program
Taxonomy. We first fix a label set from the archetypes in the file: administrative mastheads, conference front matter and indexes, colocated-event and historical listings, commemorative tributes, and tutorial or synthesis volumes. Position papers such as Stop treating AGI as the north-star are scholarly arguments rather than administrative records. Folding them into "non-research" would inflate the estimates and alienate readers, so we keep them as a separate "non-empirical scholarly" class. We report them apart from the other classes.
Classifier. Following FineWeb's recipe, an LLM labels a few tens of thousands of documents from title plus first-page text. Those labels train an embedding classifier that scores every record in S2ORC.
Audit. Humans label a stratified sample, with strata defined by field, venue type, year band, and classifier-score decile. Sampling is heavier in strata where the classifier is uncertain.
Estimator. For a stratum with $N$ records, let $f(x)$ be the classifier's probability that record $x$ is non-research. Let $y_j$ be the human label on the $n$ audited records in that stratum. The contamination estimate is
$$\hat\theta = \frac{1}{N}\sum_{i=1}^{N} f(x_i) + \frac{1}{n}\sum_{j=1}^{n}\big(y_j - f(x_j)\big).$$
The first term is the classifier's guess. The second term corrects it by the classifier's average error measured on audited records. The estimate is unbiased whatever the classifier's errors. Its variance falls as the classifier improves, because the residuals $y_j - f(x_j)$ shrink. This property is what lets a small audit support corpus-wide claims.
External ground truth. As a second, independent check we use venues whose accepted-paper lists are separately known. ACL Anthology volumes reserve dedicated entries for front matter and tutorial volumes, such as C18-3000. VLDB issues publish their accepted-paper rosters in the front matter itself. For these venue-years, the true non-research count is simply indexed records minus accepted papers. That gives a Strava-versus-city-count style validation at scale.
Downstream effects. We measure two consequences. For bibliometrics, we recompute citation counts, h-indices, and Content-Informed Index weights on the S2ORC citation graph with and without non-research nodes, and record how many venue and author rankings change. For extraction, we run MOLE's schema on a held-out set of non-research documents and count fields filled with confident but fabricated values, such as a "dataset volume" for a masthead.
Baselines
Metadata tags. The naive reference is to trust the database type field from OpenAlex or DBLP and count records not tagged as research. The documented mislabelling of mastheads and front matter suggests this will undercount badly. It is still the default most corpus builders use, so every estimate is reported against it on identical strata.
Title rules. The second baseline is regular expressions over titles, for patterns such as "Front Matter", "publication information", "Organizing Committee", and "Remembering". It costs almost nothing and should be very precise. On the full corpus we report its classify-and-count prevalence. On the audited sample we report its recall separately for each archetype.
Distilled classifier with plain counting. The strongest competitor is the same FineWeb-style classifier with counts taken at face value. It shares the corpus, the embedding model, the LLM labelling budget, and the hyperparameter search with our method. It differs only in lacking the audit correction. This is therefore also the ablation that isolates our key ingredient.
A second ablation replaces the stratified audit with a uniform random audit of the same size. This tests whether targeting uncertain strata matters. As a ceiling on per-document accuracy, we run the LLM judge directly on the audited records only.
There is no reported result on Lacuna for non-research document detection or contamination prevalence, so no prior number exists to beat on the core task. For the downstream extraction comparison, the reference is MOLE's top average F1 of 75.23 with Gemini 1.5 Pro on the MOLE Benchmark (MOLE). We expect that number to hold on genuine papers. We report alongside it a fabrication rate on non-research inputs, which MOLE never measured.
What counts as success. Our estimator must achieve lower absolute error than plain counting against registry-derived counts, measured across venue-years. Its 95% intervals must cover the held-out audit prevalence close to nominally. Its intervals must also be narrower than those of an audit-only estimate of the same size.
Why not just…?
Why not just use the database type field? The file documents mastheads and front matter tagged research_article and a tutorial-abstracts volume tagged conference_paper in OpenAlex, DBLP, and the ACL Anthology. We still run this baseline. Its measured gap from the audited estimate is itself a headline result, because it quantifies how wrong current corpus statistics are.
Why not just regex the titles and be done? This is the real threat. If mastheads and front matter make up nearly all non-research records by count, title rules capture most of the contamination. In that case the bibliometric distortion they leave behind may be negligible. The stratified audit settles this directly by reporting, per field, what fraction of non-research mass the rules miss. Titles like From algebraic geometry to machine learning, a tribute, or A unified approach to sparse signal processing, a synthesis, suggest the misses concentrate in mathematics and in society journals. If the residual is small everywhere, we will say so and recommend regex plus a small audit. That would still be a useful, validated finding.
Why not just distil an LLM judge and count its predictions? That is our third baseline, and the classifier may be very accurate overall. The trouble is that counting predictions inherits whatever errors differ between strata, and nothing in plain counting reveals them. The registry comparison and the audit residuals show whether the correction changes stratum-level estimates by a meaningful amount. If it does not, the classifier alone is sufficient and the audit cost is unnecessary.
Why not just prompt an LLM on every document? At S2ORC's 81.1M records this means hundreds of billions of input tokens. FineWeb made LLM judging tractable only by distilling it. A direct judge would also still need an audit to establish its own error rates. We use it only on the audited subset, as a ceiling.
Decisive Checks
Four results decide the project. First, at registry-validated venue-years, corrected estimates should align with true counts better than raw classifier counts, measured by R² and absolute error. Second, interval coverage should hold up in tribute-heavy strata, where we expect the classifier to be weakest. Third, the bibliometric experiment should show whether removing estimated non-research nodes changes any top-k venue or author ranking under raw counts or under Content-Informed Index weights. Fourth, the MOLE fabrication rate on non-research inputs should be reported.
If contamination turns out small and inconsequential everywhere, that is a publishable negative result that lets corpus builders stop worrying. If it is concentrated in specific fields, we will have the first audited map of where it lives.
References
- Participatory Epidemiology: Use of Mobile Phones for Community-Based Health Reporting
- A Street-Specific Analysis of Level of Traffic Stress Trends in Strava Bicycle Ridership and its Implications for Low-Stress Bicycling Routes in Toronto
- Image-Based Social Sensing: Combining AI and the Crowd to Mine Policy-Adherence Indicators from Twitter
- #FoodPorn: Obesity Patterns in Culinary Interactions
- S2ORC: The Semantic Scholar Open Research Corpus
- Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics
- MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- IEEE Transactions on Cybernetics publication information
- Front Matter
- Proceedings of the 27th International Conference on Computational Linguistics: Tutorial Abstracts
- Position: Stop treating `AGI' as the north-star goal of AI research
- From algebraic geometry to machine learning
- A Unified Approach to Sparse Signal Processing