When a researcher types a claim into a literature search engine, they need the papers that support it and also the papers that contradict it. Modern search engines for science rank papers by comparing vectors: each claim and each abstract becomes a point in space, and nearby points are returned first. This proposal asks a simple question. Within a single research topic, do these vector spaces sit closer to papers reporting the majority direction of a finding ("drug X lowers Y") than to papers reporting the minority direction ("drug X has no effect on Y")? We do not know. If they do, retrieval quietly removes dissent before any downstream fact-checker or summarizer sees it. The stakes are concrete. In an open corpus of 500,000 biomedical abstracts, roughly 20% of well-studied claims have both supporting and refuting evidence, and verification systems lose at least 15 F1 points relative to curated settings (SciFact-Open).
We propose a consensus association test. It adapts the Word Embedding Association Test, a statistic that checks whether one set of vectors sits systematically closer to one set of attribute vectors than to another, from social bias to scientific findings. Three changes make it work for this purpose. The attribute sets are no longer hand-picked word lists; they are majority-direction and minority-direction abstracts identified automatically by distant supervision, meaning training labels produced by aligning text with structure (here, shared entity pairs and contradiction detection) rather than by human annotators. The comparison runs inside topic clusters found by an embedding topic model, so "majority" is defined locally and topical similarity cannot masquerade as bias. Finally, the test is validated by whether it predicts each retriever's recall of refuting evidence on SciFact-Open, and then used to select training data that repairs the bias. The core claim is falsifiable: a label-light geometric test, computed at corpus scale, ranks retrievers by how much contradicting evidence they lose, and training against what it flags recovers that evidence.
Where this sits relative to prior work
Two papers motivate the problem. SciFact-Open scales verification to 500k abstracts and documents both the 20% conflicting-evidence rate and the low overlap between models' retrieved candidates (Jaccard below 0.2). It measures the damage but does not diagnose which representational property causes it. GLIMPSE shows that summarizers of peer reviews suppress divergent opinions and corrects this at the summarization stage; we look one step earlier, at retrieval, where a missing paper cannot be recovered later. Do Neural Language Models Overcome Reporting Bias? finds that language models amplify the skew of their training text and are blind to negation. That is the mechanism we hypothesize, but it was shown on commonsense frequencies, not scientific retrieval.
The measurement method comes from association testing. Quantifying 60 Years of Gender Bias in Biomedical Research with Word Embeddings applies three such tests (WEAT, ECT, RIPA) to PubMed embeddings. It also states the two weaknesses we must avoid: sensitivity to curated word lists, and the inability to separate genuine domain regularities from bias. We address the first by generating stimulus sets through distant supervision. We address the second by controlling for lexical overlap and topic, so that a residual preference for majority-direction abstracts cannot be explained by those abstracts simply being more on-topic.
The data-generation method comes from Generating Scientific Claims for Zero-Shot Scientific Fact Checking. That work turns citation sentences into atomic claims, creates refutations by swapping in sibling concepts from the UMLS medical ontology, and filters candidates with a RoBERTa contradiction model. We reuse its claim extraction and contradiction filtering. The difference is in what gets labeled. Instead of synthesizing artificial negatives, we pair real abstracts that make opposing claims about the same entity pair, so the minority pole consists of findings that actually exist in the literature. The topic layer uses the standard modular pipeline described in the file (transformer embeddings, UMAP reduction, HDBSCAN density clustering, c-TF-IDF keywords). We deliberately keep HDBSCAN's "noise" documents as a separate stratum, because sparse regions may be where dissent concentrates.
Core Claim
Fix a topic cluster $t$ and a set of claim queries $Q_t$ drawn from it. Let $A_t$ be abstracts whose extracted finding agrees with the cluster's majority direction for that entity pair, and $B_t$ those that disagree. Each abstract in $B_t$ is matched to one in $A_t$ with the same BM25 score bin against the query, so lexical overlap is held fixed. With $s(q,d)$ the retriever's similarity score, the consensus association effect size is
$$d_t = \frac{\operatorname{mean}_{q, a \in A_t} s(q,a) - \operatorname{mean}_{q, b \in B_t} s(q,b)}{\operatorname{std}_{q, x \in A_t \cup B_t} s(q,x)},$$
with significance by permutation of the A/B labels, as in WEAT. A retriever's overall score is the document-weighted mean of $d_t$ across clusters. We claim three things. First, $d_t$ is positive and significant for common dense scientific retrievers. Second, its magnitude predicts loss of REFUTES-evidence recall on SciFact-Open. Third, contrastive fine-tuning that promotes matched minority-direction abstracts to positives in high-$d_t$ clusters reduces both the association and the recall loss.
Research Program
The corpus is the S2ORC-derived 500k-abstract pool underlying SciFact-Open, with the SciFact claims (Fact or Fiction) excluded from stimulus generation to prevent leakage.
The first step builds topic clusters. We fit the embedding topic pipeline with an encoder that is not among the retrievers under test, to avoid circularity. We repeat clustering over five seeds and two minimum-cluster-size settings, and keep only clusters that are stable under re-runs, measured by matched-cluster Jaccard. This responds directly to the known run-to-run instability of UMAP and HDBSCAN.
The second step constructs stimuli by distant supervision. Within each cluster, we extract entities with scispaCy and atomic claims with the CLAIMGEN pipeline. We then group abstracts by shared entity pair, which is the classic alignment assumption: abstracts mentioning the same pair are taken to discuss the same relationship. Pairs of claims are labeled agree or contradict by the contradiction model. Majority and minority poles come from the resulting agreement graph. This step is noisy. We therefore audit 300 sampled pairs with domain annotators and report pole precision. As an external check, we confirm that the 20% of SciFact-Open claims annotated as having conflicting evidence fall into clusters where our pipeline also finds two poles.
The third step validates the test against outcomes. We compute the test for roughly eight retrievers spanning BM25, MedCPT (the encoder used by Enabling Inclusive Systematic Reviews), general dense retrievers, and SciBERT-based bi-encoders. For each, we measure recall@k of SUPPORTS and REFUTES evidence separately on SciFact-Open. Eight models give little statistical power for a cross-model correlation. We therefore also test the relationship within each model across topic clusters, which provides hundreds of data points.
The fourth step repairs the bias. In clusters with high $d_t$, we fine-tune the retriever contrastively. Matched minority-direction abstracts become additional positives for queries in that cluster, and majority-direction abstracts with similar BM25 scores remain positives as well, so the model is not taught to prefer dissent. We then re-run the test and the SciFact-Open evaluation, both alone and with MultiVerS as the downstream verifier.
Baselines
The naive reference point is each off-the-shelf retriever with no intervention, together with BM25. BM25 should show near-zero association once lexical overlap is matched. That makes it the null calibration for the test: if BM25 shows a large $d_t$, the matching has failed.
The strongest existing end-to-end reference on this question is MultiVerS on SciFact-Open, which SciFact-Open reports at 52.4 F1, down from 72.5 on the curated set. We run MultiVerS unchanged over our retrievers' top-k with identical k. Beating it means a statistically significant gain in REFUTES recall without a drop in SUPPORTS recall, and a net F1 above 52.4 under the same judged pool. No reported result was found on Lacuna for retrieval recall split by label on SciFact-Open, so we establish those numbers ourselves.
The strongest training-data competitor is CLAIMGEN plus KBIN synthetic negatives (Generating Scientific Claims), which reaches 71.08 F1 on SciFact against 77.70 with human labels. We train the same retriever on the same number of examples, with the same compute and hyperparameter search budget, using KBIN-style data in place of our pole-matched pairs.
Reranking is the strongest system competitor. monoT5 reranking improves abstract retrieval by about 17 points over BM25 in VERT5ERINI, whose full pipeline scores 65.07 F1 on SciFact. We apply it on top of every first-stage retriever.
Two ablations isolate the proposal's key ingredients. The first replaces topic-local poles with global majority directions, which tests whether topic conditioning matters. The second replaces distantly supervised poles with a hand-curated list of direction verbs (increase/decrease/no effect), applied in classic WEAT fashion, which tests whether automatically generated stimuli matter.
Why not just…?
Why not just measure REFUTES recall on SciFact-Open directly and skip the geometric test? We will measure it; it is the validation target. But SciFact-Open has only 279 claims in one biomedical slice, and its judgments come from pooling existing models' outputs, so a retriever that finds new evidence is partly scored against unjudged documents. The test needs no verdict labels beyond distant ones, runs on every topic cluster, and can be computed for a new domain or encoder before any benchmark exists. If the test fails to predict recall on SciFact-Open, it has no value, and the within-model correlation across clusters is the experiment that decides this. To limit pooling bias, we annotate a sample of previously unjudged top-ranked abstracts.
Why not just add a cross-encoder or LLM reranker, which reads claim and abstract together and should handle direction? Reranking only reorders what first-stage retrieval returns. If minority-direction abstracts fall outside the top-k, no reranker can recover them. The low cross-model overlap in SciFact-Open suggests candidate sets differ substantially at this first stage. Readers are also not immune to direction: Fact or Fiction documents failures on claims that differ only in polarity. This alternative is a real threat when k is large. We therefore report REFUTES recall as a function of k with and without monoT5. If reranking at k=100 closes the gap, the bias is mainly a ranking-depth issue, and that result would narrow our claim.
Why not just train on KBIN-style synthetic negations? Ontology-sibling swaps teach the model to reject a wrong entity, not to retrieve a real paper reporting the opposite result. They also depend on UMLS, as Generating Scientific Claims acknowledges. Our pole pairs need only entity co-occurrence and a contradiction model. The matched-budget comparison in the Baselines section settles which training data improves REFUTES recall more.
Why not just use nearest neighbors instead of topic clusters to define "majority"? Neighborhoods defined by the retriever under test are circular: they inherit its biases. Global majorities mix unrelated literatures. The global-pole ablation measures whether stable topic clusters add anything. If they do not, we drop them.
Decisive Checks
The proposal stands or falls on four results. First, BM25 must show $d_t$ near zero after overlap matching, while at least some dense retrievers show significant positive $d_t$. Second, within models, $d_t$ across clusters must negatively correlate with REFUTES recall. We preregister a Spearman correlation of $-0.3$ or stronger, and require the effect to persist after controlling for cluster size and the minority fraction. Third, pole-matched fine-tuning must beat both the KBIN-trained and reranked baselines on REFUTES recall at matched k without lowering SUPPORTS recall. Fourth, the HDBSCAN noise stratum must be reported separately, to test whether dissenting findings concentrate where topic models usually discard documents. A null result on the second check would show that geometric association in scientific retrievers is a curiosity rather than a cause of lost evidence, and that conclusion would also be worth publishing.
References
- SciFact-Open: Towards open-domain scientific claim verification
- GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews
- Do Neural Language Models Overcome Reporting Bias?
- Quantifying 60 Years of Gender Bias in Biomedical Research with Word Embeddings
- Generating Scientific Claims for Zero-Shot Scientific Fact Checking
- Fact or Fiction: Verifying Scientific Claims
- Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations
- Scientific Claim Verification with VerT5erini