Do Paper Filters Detect Missing Science, or Just Masthead Words?

Scholarly indexes contain many records that are not research papers: journal "publication information" pages, conference front matter, memorial tributes, lists of past conferences. They are often indexed as research articles with editorial board members listed as authors. Whatever filter removes them, whether a regex, a small text classifier, or a large language model (LLM) judge, should remove them for the right reason. The right reason is that the document contains no argument, no method, and no citations tied to claims. A filter that instead reacts to surface words like "Editorial Board" or "in memoriam" will mishandle the cases that matter most: position papers, tutorial abstracts, and tributes that discuss real mathematics.

This note proposes a benchmark and a test protocol that separate these two kinds of filter. We give each filter controlled edits to its input. Some edits remove or insert keywords while leaving document structure unchanged. Others remove or insert structure, such as a bibliography with inline citations or the repeated name-and-affiliation layout, while leaving keywords unchanged. We then measure which kind of edit flips the filter's decision. A recent result on black-box language models as nondeterministic causal models justifies reading these flip rates as causal effects, even for stochastic LLM judges we can only query. We also test a cheap structural signal: how fast the number of distinct substrings grows with substring length. This measure comes from combinatorics on words, and we ask whether it gives a filter that is both inexpensive and caused by structure rather than vocabulary.

Core Claim

Accuracy on a held-out set cannot tell a structure-driven filter from a keyword-driven one, because in natural data the two kinds of cue almost always appear together. Edit-based causal testing can tell them apart. We claim, as a hypothesis to be tested, that keyword-trained classifiers and zero-shot LLM judges will be markedly more sensitive to keyword edits than to structure edits. We also expect a filter built on substring-growth profiles plus citation-link structure to reverse that ratio, and to hold up better on boundary genres, at a cost close to that of a linear classifier. If the cheap supervised classifier turns out to be just as structure-driven, the benchmark and protocol remain the contribution and the complexity features are not needed. That outcome is settled by experiment, not argued away.

Background and How Each Direction Is Used

The application comes from the record of non-research artifacts in the index. Examples include IEEE Transactions on Cybernetics publication information, Front Matter, A memorial tribute: Adriano Garsia (1928--2024), Past IJCAI Conferences, and the boundary-genre Position: Stop treating `AGI' as the north-star goal of AI research. These documents motivate the problem. They also show that existing pipelines let such records through. S2ORC parses 81.1M records and links inline citations to bibliography entries, but it does not reject non-papers. DocGenome classifies arXiv papers by discipline and largely excludes administrative material before annotation. Neither offers a labeled corpus for this filtering task, and we build one.

The causal method comes from Large Language Models as Nondeterministic Causal Models. That paper treats the prompt as the only root cause of a stochastic output and proves that the observational, interventional, and counterfactual distributions coincide. Re-sampling an LLM on an edited input is therefore a valid counterfactual query. We apply this to LLM-based filters, where the alternative, fixing the model's random noise, is unavailable behind an API. The same paper notes that its semantics do not guarantee that a counterfactual output stays close to the original. We avoid depending on that property by measuring flip probabilities over many samples rather than single outputs. Beckers' companion work, Actual Causation and Nondeterministic Causal Models, defines an actual cause as counterfactual dependence in a simplified model. We use it only to frame the question of whether "absence of citation structure" is an actual cause of a "non-research" verdict. We do not use its full exhaustive-query machinery, which the file notes is intractable for large domains. The SMT-based approach of Efficient Discovery of Actual Causality in Stochastic Systems is restricted to small Markov chains over control tasks and does not carry over to text classifiers, so we do not build on it.

The structural signal comes from combinatorics on words. Block complexity $p(n)$, the number of distinct length-$n$ substrings, rises quickly toward its ceiling for running prose. For templated lists of names and affiliations it stays far below that ceiling. Joint string complexity for Markov sources shows that the number of substrings shared between two texts grows linearly when they come from the same source and as $n^\kappa$ with $\kappa<1$ otherwise. This gives a distance $d(X,Y)=1-\log J(X,Y)/\log n$, where $J$ is the shared-substring count and $n$ the text length. The paper warns that its Markov assumptions do not hold for real language. We use $d$ only as an empirical feature against reference corpora of research prose and of boilerplate, not as a calibrated test.

The squarefree base requirement plays a narrow role, and we say so plainly. Complexity of Leading Digit Sequences proves two things about the sequence of leading digits of $a^n$ in base $b$. For squarefree $b\ge5$, its complexity is exactly affine, $p_{a,b}(n)=c_{a,b}n+d_{a,b}$. For non-squarefree $b$, some $a$ give non-affine complexity. These sequences give us a source of exact ground truth for our complexity-curve estimator. Before applying the estimator to text, we check that, on finite prefixes, it recovers the known slope and correctly flags non-affine growth. We reject the file's suggested link between squarefreeness and discretizing causal abstractions. Nothing in the material supports it, and adding it would be ornament.

Research Program

We build three things. The first is a labeled benchmark of index records with three classes: administrative, semi-research (position papers, tutorial volumes, tributes, event histories), and research. Negatives are drawn from venue records of the kinds listed above. Research positives come from S2ORC full text and DocGenome. Labels come from two annotators with adjudication.

The second is an edit suite applied to held-out documents. Keyword edits insert scientific discourse markers ("we propose", "baseline", "statistically significant") into mastheads, or delete administrative headers, while leaving layout and citation structure alone. Structure edits remove the bibliography and inline citation markers from real papers, or graft a citation-linked bibliography and a prose paragraph onto front matter, while holding the keyword inventory fixed as far as possible. A small human audit checks that edited documents are still realistic, and we report how often they are not.

The third is the structural filter. It is a small model over three groups of features: the normalized block-complexity curve $p(n)/(L-n+1)$ for character and token windows, where $L$ is the document length; joint-complexity distances to the two reference corpora; and S2ORC-style citation-link counts. It deliberately has no access to word identities beyond what these statistics encode.

For any filter $F$ and edit family $E$, the main quantity is the flip rate, the probability that $F$'s verdict changes when $E$ is applied, averaged over documents and, for stochastic judges, over samples. Under Beckers' semantics this flip rate is the interventional effect of the edit. The structure-reliance ratio is the structure-edit flip rate divided by the keyword-edit flip rate. A ratio well above one means the filter's decisions are caused by what the file calls structural absence.

Baselines

The naive reference point is to trust the index descriptor, which marks every record as a research article. It removes nothing and sets the floor for precision of the research class.

The keyword baseline is a hand-written regex filter over administrative phrases and metadata cues ("Editorial Board", "ISSN", "Program Committee", "in memoriam"). We write it before seeing the test set, with the same development data as all other methods. It counts as beaten if another method wins on boundary-genre F1 and on structure-reliance ratio.

The strongest cheap competitor is a supervised text classifier. We train both a TF-IDF logistic regression and a fine-tuned scientific BERT encoder of the kind S2ORC-SciBERT represents. Both use the same training split and the same hyperparameter-search budget as our structural filter. No reported result was found on Lacuna for any classifier on non-research document filtering, so this baseline has no external number.

The strongest existing method is a zero-shot LLM judge. We use GPT-4o, the model DocGenome evaluates, prompted with class definitions and sampled several times per document. The closest reported figure is GPT-4o's 97.6% document classification accuracy on DocGenome-test (DocGenome). That figure is for discipline classification of genuine arXiv papers, not for non-research filtering, so it shows the model's general strength but is not a direct target. Beating the judge means matching its macro-F1 within a few points, exceeding its structure-reliance ratio, and doing both at far lower cost per million documents.

Two ablations isolate the key ingredient. One removes the complexity-curve and joint-complexity features, leaving only citation counts. The other removes citation counts, leaving only complexity features. Together they show whether substring-growth profiles add anything beyond the obvious citation signal.

Why not just…?

Why not just use a zero-shot LLM, since GPT-4o already classifies documents at 97.6%? That number comes from a different task on clean arXiv papers. Even if the judge is accurate here, running it over the 81.1M records of an S2ORC-scale index is expensive, and accuracy alone says nothing about why it rejects a document. The edit suite answers that directly. If the judge turns out to be both accurate and structure-driven, it becomes the reference standard, and our cheap filter is judged by how closely it matches the judge at a fraction of the cost. That would be a narrower but honest result.

Why not just use keyword rules on metadata? Tributes and position papers discuss methods and cite researchers, so keyword cues overlap across classes. The persistence of mastheads indexed as articles shows that existing pipelines do not catch them. Still, rules may do fine on pure mastheads. The benchmark reports per-class results, so a rule set that solves the administrative class but fails on semi-research genres will be visible as exactly that.

Why not just fine-tune a small text classifier, which is cheap and probably accurate? This is the real threat. A TF-IDF or SciBERT classifier may match our filter on held-out F1 at similar cost. Our claim survives only if it is markedly more sensitive to keyword edits than to structure edits, or if it degrades on boundary genres. If its structure-reliance ratio matches ours, the complexity features do not earn their place. We would then publish the benchmark and protocol, report that simple supervision already learns structure, and drop the complexity-feature claim.

Why bring in the squarefree theorem at all? Only as a check on the estimator, since leading-digit sequences have provably exact complexity. If the estimator is validated more easily by other means, this part can be dropped without affecting the main claim.

Proposed Experiments

The first experiment is the estimator check. For squarefree bases such as 6, 10, and 15 and admissible rational $a$, we compare estimated slopes on prefixes of increasing length with $c_{a,b}$. We also confirm that the estimated slopes lie within the bounds $\tfrac{b-1}{2}n+\tfrac{b-1}{2}\le p_{a,b}(n)\le(b-1)n$ proven by He and Hildebrand. For non-squarefree bases such as 8 and 12, we test whether a deviation-from-affine statistic separates the non-affine cases that their Theorem 2.3 guarantees. Separately, we reproduce the qualitative English-versus-other-language gap in joint complexity that Jacquet et al. report, $\kappa\approx0.18$ for French and $\kappa\approx0.1$ for Polish, as a sanity check on our joint-complexity code.

The main experiment trains every baseline and our filter on the benchmark's training split. We report macro-F1, per-class F1 with emphasis on the semi-research class, research-class precision at fixed recall, and wall-clock cost per million documents.

The decisive experiment runs all filters on the edit suite and compares structure-reliance ratios with bootstrap confidence intervals over documents. For the LLM judge, we take at least ten samples per edited input so that flip rates estimate Beckers-style counterfactual probabilities rather than single noisy outcomes. The hypothesis fails if the supervised classifier's ratio lies within our filter's interval, or if our filter's ratio does not exceed one. It is supported if our filter reaches a ratio clearly above one, keyword-based methods stay below one, and our filter's macro-F1 is within a few points of the LLM judge at linear-classifier cost.

A final transfer test trains on IEEE and ACM records and evaluates on a held-out venue family. This checks whether structure-driven filters degrade less than keyword-driven ones when administrative vocabulary changes.