Do Scholarly-Corpus Filters Read Structure or Boilerplate?

Digital libraries and the corpora built from them index everything a publisher releases. That includes journal mastheads, committee rosters, tables of contents, obituaries, tutorial abstracts and position papers, alongside actual research articles. Any pipeline that mines "papers" for claims, datasets or ablations has to filter these items out first. Pretraining corpus builders, citation-graph builders, and automated-science tools such as the ablation planners studied in AblationBench, which assumes every input is an empirical paper, all depend on that filter. Filters built from a bag of words, a fine-tuned scientific encoder, or a single prompt to a large language model score well on the easy cases. Nobody knows why they decide what they decide. A filter that has learned "contains the string ISSN or Associate Editor" will look accurate. It will then quietly discard a memorial essay that has references, or keep a masthead that a PDF parser merged into the first article of an issue.

We propose to answer the "why" by direct intervention rather than by reading explanations. We build controlled edited documents in which one structural component is added, removed or replaced while everything else stays fixed. Examples are the administrative boilerplate, the reference list, the author list, and the methods-and-results sections. We then measure how much each component actually moves each filter's decision. This per-component effect profile does two jobs. It is the ground truth for scoring the explanations filters give for themselves, whether free-text rationales or token attributions. It is also, we hypothesize, a better predictor of how a filter will fail on hard "grey-zone" documents than its ordinary test accuracy.

Core Claim

A filter's measured dependence on boilerplate, relative to its dependence on the presence or absence of an empirical contribution, predicts its false-exclusion and false-inclusion rates on grey-zone documents better than in-distribution accuracy does. We call this ratio the shortcut index. Grey-zone documents are position papers, tutorials, tributes, and parser-merged front matter. The same edited-document set also shows whether language-model rationales name the components that actually drive the decision. We expect them often not to, which is the plausibility trap that NeuroFaith documents for self-explanations on short reasoning tasks.

The contribution is a measurement instrument and a tested prediction. It is not a new classifier. What would be new is showing that component-level causal effects, measured cheaply on edited documents, forecast failures that aggregate accuracy hides, in a domain where filtering errors propagate silently into downstream science tools.

Relation to Prior Work

Three papers supply the method. CEBaB evaluates concept explainers on restaurant reviews. It compares each explainer's predicted effect against the true change in model output under human-written counterfactual edits, which it calls the individual causal concept effect. We reuse that design. Our concepts are document-structure components of multi-page scholarly items instead of sentiment aspects of short reviews. CEBaB also constrains us: its naive "Approximate Counterfactual" baseline, which samples a real training example with the target concept label, beat TCAV, ConceptSHAP and INLP. Any explainer we test must therefore face that baseline. Blindfold Baselines for Embodied QA showed that a text-only bag of words reached 50.34% accuracy on EQAv1 and beat full agents at long horizons. That result motivates our insistence that lexical-only filters be treated as serious contenders, not strawmen. The pixel-flipping tradition from Binder et al. supplies the deletion-based faithfulness check we apply to token attributions.

Two papers motivate the concern that neural rationales cannot be taken at face value. NeuroFaith finds that stated concepts in language-model explanations often are not causally active inside the model, and that scale does not fix this. It studies Wikidata two-hop questions and short topic classification, not long documents. Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis shows that automated verifiers disagree on unsupported-citation rates by a range of 3% to 18%, with negative-specific agreement of only 0.27 to 0.30. It also supplies the split-conformal guard we adapt to bound false exclusion. That paper treats sources as retrievable passages and assumes they are all research. We instead treat the source document's own structure as the object under audit.

The scientific-document classification work constrains our choice of models. WANDER and Structural Scaffolds both show through ablation that structural signals help topic and citation-intent classification. Neither tests documents that lack scientific sections altogether. S2ORC documents that GROBID-style parsing merges front matter into articles, so our edits must also be run under realistic parse noise. The broader worry that gains get attributed to the wrong component, raised by Lipton and Steinhardt and Sculley et al., is the component-contribution lens we turn on the filters themselves.

Research Program

The first step is a labeled pool. We sample items from S2ORC full texts and from the kinds of records visible in scholarly indexes, and label each for one property: whether it carries an original empirical or technical contribution. The pool is stratified into three kinds of item. Clear non-research items include mastheads such as IEEE Transactions on Cybernetics publication information, committee lists, and front matter. Clear research articles form the second stratum. The third stratum is a deliberately large grey zone. It contains position papers like Position: Stop treating `AGI' as the north-star goal of AI research, tutorial abstracts, memorial essays like A memorial tribute: Adriano Garsia (1928--2024), short theory notes without experiments, and GROBID outputs where front matter was merged into a real article. Two annotators label each grey-zone item, and disagreements are adjudicated.

The second step is the edited-document set. We apply five edit families.

  • Boilerplate injection inserts copyright, ISSN and editor-list text into research articles. It should not change the label.
  • Boilerplate stripping removes the administrative lexicon from front matter. It should not change the label.
  • Reference-list deletion or insertion should not change the label.
  • Author-list inflation or removal should not change the label.
  • Contribution removal deletes the methods, results and experiment sections of a research article and leaves introduction, discussion and references. The resulting label is decided by annotators.

Humans check a sample of every edit family for fluency and label correctness. The purpose is to avoid the problem of edits that look out-of-distribution rather than like a genuine component change. For each filter $f$, component $c$, and edited pair $(x, x_{c'})$, we record the effect $f(x_{c'}) - f(x)$ on the predicted probability of "research". We summarize each filter by its shortcut index. The index is the mean absolute effect of the label-preserving lexical edits divided by the mean absolute effect of contribution removal. A value near zero means the filter reads substance, and a large value means it reads boilerplate.

The third step scores explanations against these measured effects. There are three kinds of explanation. The first is free-text rationales from prompted language models. We extract which components each rationale cites and check whether the component with the largest measured effect is among them. The second is token attributions from integrated-gradients-style methods. We aggregate attributions to the component level and compare the resulting ranking with the ranking of measured effects. The third is the Approximate Counterfactual estimator from CEBaB, applied here by retrieving a real corpus document that differs in the target component.

The fourth step adapts the split-conformal guard to calibrate each filter's exclusion threshold. The guarantee is that at most a fraction $\alpha$ of genuine research items are discarded. We then check whether the guarantee survives on the grey zone, which is by construction a distribution shift from the calibration set.

Baselines

The naive reference point has two parts: a filter that keeps everything, and a rule-based filter. The rules use title strings (Front Matter, Committee, Publication Information, Preface, Tutorial Abstracts), a zero-reference check, and an author-count threshold. These are the rules a curator would write in an afternoon. They define the floor on both the clear strata and the grey zone. Any learned filter that fails to beat them on the grey zone has no reason to exist.

The lexical contender is a TF-IDF logistic regression and a bag-of-words classifier trained on the same labeled pool. This is the analogue of the blindfold baseline and the most likely filter to have learned pure boilerplate shortcuts. It gets the same training data and the same held-out splits as every other filter. Its hyperparameters are tuned by the same number of validation trials we give the neural filters.

The strongest learned filters are S2ORC-SciBERT from S2ORC, fine-tuned on the pool, and a single-call chain-of-thought prompt to a large language model. We give the prompt the most weight among the prompting methods because AblationBench found single-call chain-of-thought planners beat multi-step agents on a related scholarly-reasoning task. On that benchmark, the best frontier model reached an AuthorAblation F1 of 0.42 against 0.65 for PhD students. That comparison is on ablation planning, not triage. There is no reported result found on Lacuna for non-research document triage accuracy, so every filter's number here is one we measure. A filter counts as "better" only if it lowers grey-zone false exclusion at matched clear-stratum accuracy.

For scoring explanations, the strongest existing competitor is CEBaB's Approximate Counterfactual estimator. CEBaB reports that it achieved the lowest ICaCE-Error of all methods it tested. The file gives no numeric value for that error, so our bar is simply to match or beat it on the same edited pairs.

The ablation that isolates our key ingredient replaces the controlled edits with naturally occurring near-duplicate document pairs mined from the corpus. If shortcut indices computed from mined pairs predict grey-zone failures as well as those from controlled edits, the controlled construction is unnecessary.

Why not just…?

Why not just write good rules? For mastheads and committee lists, rules may well be sufficient, and we will say so if they are. The threat to rules is the grey zone. Tributes, position papers and tutorials carry authors, abstracts and references, so zero-reference and author-count checks pass them through. GROBID merging, documented in S2ORC, plants real article text inside front-matter records. The stratified evaluation will show whether rules fail there. If they do not, the proposal shrinks to an audit of learned filters that rules make unnecessary. That would still be a useful negative result.

Why not just ask a frontier model and read its rationale? Because the rationale is the thing under test. NeuroFaith shows that stated reasons often are not causally active, and the citation-verifier disagreement in Evaluating and Guarding Citation Faithfulness shows that model judgments on scholarly text are not self-certifying. The edit set tells us directly whether the prompted model's decision flips when boilerplate is injected while its rationale cites "no methodology section". If prompted models turn out to be both accurate and rationale-faithful, that is a positive finding the program is designed to report.

Why not just test on a held-out grey-zone set and skip the counterfactuals? This is the most serious alternative. A held-out test measures failure but does not say which component caused it or whether a fix targets the right thing. Held-out accuracy is also exactly the quantity our core claim says is a weak predictor. The decisive experiment below compares the two predictors head to head. If held-out accuracy on a small grey-zone development set predicts failures as well as the shortcut index, the counterfactual machinery is not earning its cost, and we would drop it in favor of simple grey-zone test sets.

Why not just use nearest-neighbor counterfactuals instead of constructed edits? CEBaB's result says this can be surprisingly good, so it is our isolating ablation, not a dismissed alternative. Mined pairs differ in many components at once, which confounds component-level effects. That is why we expect them to predict failures less sharply, but we treat this as an empirical question.

Proposed Experiments

The decisive check runs over a population of about thirty filters. The population varies architecture (TF-IDF, SciBERT, prompted language models of several sizes), training mix, and prompt wording. For each filter we compute its clear-stratum accuracy and its shortcut index on a development edit set. We then measure its false-exclusion and false-inclusion rates on the held-out grey zone and on a held-out set of parser-merged documents. The core claim holds if the shortcut index correlates with grey-zone error more strongly than clear-stratum accuracy does, with a bootstrap confidence interval that excludes equality. It fails if accuracy predicts as well.

The second check scores explanations. It asks how often the top measured component appears in free-text rationales and how well component-aggregated attributions rank-correlate with measured effects. Both are compared against the Approximate Counterfactual estimator and against a random-ranking floor. We will report whether mid-sized models give more faithful rationales than larger ones, which would replicate the NeuroFaith pattern on long documents.

The third check tests the conformal guard. We calibrate on clear strata at several values of $\alpha$ and report realized false exclusion on the clear test set and on the grey zone. We expect the guarantee to hold on the clear test set, since that is what split-conformal promises under exchangeability. We expect it to break on the grey zone, and to break most for high-shortcut filters. We then test whether recalibrating with a small grey-zone calibration set restores coverage.

The final check follows the errors downstream. Grey-zone items that each filter misroutes are fed to the single-call ablation planner setup from AblationBench. We record how often the planner invents empirical ablations for documents that have no experiments. This measures what a filtering mistake costs the automated-science tools that motivated the work.