Off the Tree or Between Branches? Placement-Style Scores for Non-Research Documents

Scholarly indexes and pretraining corpora treat every record as a research paper. Many records are not. They include journal mastheads, proceedings front matter, committee rosters, calls for papers, memorial tributes and tutorial abstracts. Flat classifiers handle these records badly. Asked which field a masthead for a fuzzy-systems journal belongs to, a classifier answers "fuzzy systems" with confidence. Asked about a genuinely interdisciplinary paper, it spreads its probability across fields. A single uncertainty number such as softmax entropy cannot tell these two failures apart, and the first one is the one that pollutes corpora.

Biology has already solved a structurally similar problem. Phylogenetic placement attaches a short, unknown DNA read to a fixed reference tree of known species. The result is not only the most likely branch but also two separate quantities. The first is how far off the tree the read hangs, its pendant branch length. The second is how widely its probability is spread across branches, measured by the expected distance between placement locations (EDPL) of pplacer. We propose to place document embeddings onto a reference tree of research fields in the same way. The test is whether these two numbers separate "not research at all" (far off the tree) from "research between fields" (close to the tree but spread across distant branches). The scores need no labeled examples of non-research documents.

Core Claim

A document's position relative to a tree built only from research papers gives two signals that are close to independent. Administrative and commemorative texts should show long pendant lengths. Their mass should often be concentrated near their venue's field, because a masthead borrows the field's vocabulary but is unlike any paper in it. Interdisciplinary research should show short pendant lengths and high EDPL. Non-empirical scholarship, meaning position papers, tutorials and reviews, should sit in between. If this holds, the two-dimensional score should beat scalar open-world scores at the hard boundary: keeping interdisciplinary and position papers while removing front matter and tributes. It should also generalize to non-research genres never seen during training, which a supervised filter cannot promise.

The research file reports no prior work applying placement to text (pplacer and Edge PCA and squash clustering were built only for sequence alignments). Tree-aware text and sequence classifiers exist, but they do not address this question. Phylo-HS builds a phylogeny into a hierarchical softmax, and graph-based document classifiers such as D2GCLF model links between documents. Both assume every input belongs at some leaf, and neither produces an off-tree distance. Our design differs from them in exactly that respect.

Research Program

The reference tree has research subfields as leaves. We will build it in two ways and compare them: from an existing field-of-study hierarchy over S2ORC metadata, and by hierarchical clustering of subfield centroids. Documents are represented by SciBERT embeddings of title, abstract and the first page of text, reduced by PCA to a few hundred dimensions.

We treat embeddings as continuous traits that drift by Brownian motion along the tree. Under that model, the likelihood of attaching a query to edge $e$, at some position along it and with pendant length $\ell$, is Gaussian. Partial likelihoods can be cached per edge with the same pruning recursion that makes pplacer linear in the number of reference taxa. Per-clade variances are fit only on research papers. The tree therefore normalizes distance by how spread out each region of science actually is, and small clades borrow strength from their ancestors.

For each query we keep three outputs. The first is the posterior mass $p_e$ over edges. The second is the maximum-likelihood pendant length $\ell^*$. The third is the EDPL:

$$\mathrm{EDPL} = \sum_{e,f} p_e\, p_f\, d(e,f),$$

where $d(e,f)$ is the path length between edges $e$ and $f$ on the tree. The filtering score is a fixed rule in the $(\ell^*, \mathrm{EDPL})$ plane, tuned on research papers alone. Concretely, we flag a document when $\ell^*$ exceeds a high quantile of research pendant lengths, conditioned on placement region.

A secondary use is auditing a whole venue. We sum placement mass over each venue's records, then apply Edge PCA and the Kantorovich–Rubinstein (earth-mover) distance from Edge PCA and squash clustering. This shows where in the field tree off-tree mass accumulates, for example in VLDB proceedings full of front matter or IEEE transactions issues full of mastheads.

Baselines

The naive reference point is a title and metadata rule set: regular expressions for "Front Matter", "Publication Information", "Committee", "Call for", "Remembering" and "Tutorial Abstracts", plus page-count and missing-abstract checks. It costs nothing and will catch most mastheads. Beating it means higher recall on non-research documents at matched research recall, especially on tributes and on title-ambiguous items such as A unified approach to sparse signal processing.

The second baseline is a structural filter: drop any record where GROBID finds no abstract or method section, mirroring the S2ORC pipeline. It is the de facto practice. It is run on the same PDFs, and beating it means not discarding position papers or atypically formatted research.

The strongest label-free competitor is per-class Mahalanobis distance on the same SciBERT embeddings, together with k-nearest-neighbour distance to research papers and the field-label entropy of those neighbours. This pair captures "far from data" and "between classes" without any tree. It isolates whether the tree's topology and its hierarchical variance sharing add anything.

The strongest supervised competitor is SciBERT fine-tuned with an explicit non-research class. SciBERT's general strength is documented: it averaged 2.11 F1 points over BERT-Base across the downstream tasks in SciBERT. For non-research document detection itself, we found no reported result on Lacuna, so no figure from the literature exists to beat. This classifier gets the same embeddings, the same training budget, and hyperparameters tuned on the same validation split.

The key ablations hold everything else fixed while replacing the reference tree. One replaces it with a star tree, which removes hierarchy. Another uses a random topology over the same leaves. A third reports $\ell^*$ alone and EDPL alone.

For every baseline, beating it means higher AUPRC for non-research detection at research recall fixed at 99%. Interdisciplinary papers count as research when computing that recall.

Why not just…?

Why not just use title regexes? They will be very hard to beat on mastheads and front matter, and we expect them to remain the first stage of any real pipeline. Where they leave off is tributes with idiosyncratic titles and documents whose titles look like papers, as with A memorial tribute: Adriano Garsia and A unified approach to sparse signal processing. We will report results on the residual left after regex filtering, so any gain is measured where the regexes leave off.

Why not just fine-tune SciBERT with a "non-research" class? On genres present in its training data, it will probably match or beat us, and we will say so. Its weakness is the open world. New kinds of administrative text keep appearing, and closed-world softmax classifiers tend to be overconfident outside their training classes, the concern raised in Mitigating Uncertainty in Document Classification. The held-out-genre experiment below settles the question. If the fine-tuned model still wins on genres it never saw, the label-free argument fails.

Why not just use k-nearest-neighbour or Mahalanobis distance in embedding space? This is the real threat, because pendant length is close to a distance to the data. The tree should help in two ways. Its variances adapt per clade, and EDPL measures spread in tree distance, where neighbour-label entropy treats "computer vision vs. NLP" the same as "computer vision vs. ecology." If the star-tree ablation and the Mahalanobis/kNN pair match the full model, we will report that the tree adds nothing.

Proposed Experiments

No labeled benchmark for this task exists in the file, so we will build one. We will draw about 5,000 S2ORC records, stratified by venue, deliberately oversampling proceedings and IEEE transactions issues. Two annotators will label each record as empirical research, interdisciplinary research, non-empirical scholarship (position, tutorial, review), administrative (masthead, front matter, committee, call) or tribute, and we will report agreement between them. Seed exemplars come from the file: VLDB front matter, IEEE TFS publication information, DSML 2023 Committee, Remembering Don Bryant, the AGI position paper, and ACL tutorial abstracts.

The first decisive check is the plane plot. Do administrative documents and interdisciplinary papers occupy different corners of $(\ell^*, \mathrm{EDPL})$, and does AUPRC on that pair beat softmax entropy and the Mahalanobis/kNN pair? The second is held-out genres. The supervised classifier is trained on mastheads and front matter only, and every method is tested on tributes, calls and tutorials. The third tests robustness to tree misspecification, the known weakness of fixed-tree placement. We will compare the curated taxonomy, the clustered tree and a random tree, and also drop whole subfields from the reference to see whether papers from missing fields are wrongly flagged as non-research.

As a scale sanity check, pplacer places roughly 20,000 reads on a 1,000-taxon tree per hour per processor. Our Gaussian version should be cheaper per query, and we will measure its throughput on the full S2ORC metadata. Finally, the venue-level Edge PCA audit is judged by whether its top components align with venues whose contamination we annotate independently.