Curated scientific databases such as the drug-interaction and gene-function resources that biomedicine depends on cannot keep pace with the literature. A natural fix is to run several AI extractors over each paper, write what they agree on straight into the database, and send the rest to a human curator. The flaw in this plan is that "the models agree" is weaker evidence than it looks. Models built on the same pre-training data tend to make the same mistakes, so three models agreeing may amount to little more than one opinion repeated three times. We propose to score each candidate fact by how much independent agreement supports it. The score discounts votes from models whose errors are known to be correlated. That score then drives a statistically calibrated rule that decides which facts go into the database and which go to a curator, with a guaranteed error rate for each relation type.
The two-sentence version is this. Error-controlled abstention wrappers already guarantee a target error rate for whatever confidence score they are given, so the only open question is which score lets the system accept the most facts at that error rate. We claim that, for ensembles of related language models, a score built from the effective number of independent agreeing extractors beats raw agreement, single-model confidence, and entropy. We also predict that its advantage grows in proportion to how correlated the ensemble's errors are.
Core Claim
Selective prediction means letting a model abstain, and its performance is measured by two numbers. Coverage is the fraction of inputs on which the model commits to an answer. Selective risk is the error rate on those committed answers. Wrappers such as Selection with Guaranteed Risk (Selective Classification for Deep Neural Networks) choose a confidence threshold on a held-out labeled set so that test risk stays below a target $r^*$ with probability $1-\delta$. Stratified conformal selection (Act or Defer) extends this guarantee to each class separately. That matters because a guarantee that holds only on average can quietly fail on rare classes.
Both wrappers are indifferent to where the score comes from. This shifts the scientific question from whether the system has guarantees to which ranking of candidate facts yields the most coverage once the guarantee is enforced. Our claim is narrow and falsifiable. When extractors share lineage, raw vote agreement overranks facts on which correlated models fail together. Replacing the raw count with an effective count of independent votes recovers coverage at a fixed per-class risk. When the extractors happen to be nearly independent, the method should reduce to plain agreement and gain nothing. That second prediction is what separates a real mechanism from extra machinery.
The ingredients come from three bodies of work, each playing a different role. Two papers motivate the problem. Quantifying Correlations of Machine Learning Models measures model error correlations above 0.7 within shared families and shared pre-training regimes. Diversity is the strength of the AI crowd finds that combining similar frontier models gained about 0.6 Baseline Points on forecasting, compared with about 2.7 for a diverse ensemble.
Two papers supply the method. Beyond Majority Voting derives accuracy- and correlation-aware weights for combining language models. However, it uses static global weights, forced-choice multiple-choice answers, and no abstention. Subjective Logic, a formalism that represents an opinion as belief, disbelief, and an explicit "no evidence" mass $u$, provides the bookkeeping. Discounted Belief Fusion uses it to move mass into $u$ when sources confidently disagree. It reaches 1.00 AUC for detecting conflicting inputs on Caltech101, but it provides no risk guarantee and does not account for redundancy among sources that agree.
One paper constrains the design. The Act or Defer result shows that marginal conformal prediction can fall to about 10% selected coverage on a rare positive class, which would leave a 90% error rate on committed predictions. That result forces us to use per-class budgets.
Research Program
We first build a deliberately lineage-diverse pool of extractors for drug–drug interaction classification. The pool includes supervised encoder or Tree-LSTM models trained on the DDI'13 training split, in the style of Lim et al.. It also includes open instruction-tuned models that Comparative Performance Evaluation of LLMs for Molecular Interactions found strong on biomedical relations (Mixtral-8x7B-Instruct, LLaMA-2-Chat 70B, MPT-Chat, SOLAR-Instruct), along with domain models such as BioMedLM and Galactica. Several seeds or fine-tunes of a single base model are added on purpose, so that the pool contains known redundancy.
For each candidate drug pair, each extractor emits a label and a confidence. On a labeled calibration split, we estimate the pairwise error correlation $\rho_{mk}$ between extractors $m$ and $k$ using the phi coefficient. Each correlation is estimated per relation type and shrunk toward the global value, because calibration labels are scarce for rare types.
For a candidate label $c$ supported by the set $S_c$ of extractors with confidence weights $w_m$, the effective evidence is
$$n_c = \frac{\left(\sum_{m \in S_c} w_m\right)^2}{\sum_{m,k \in S_c} w_m w_k \rho_{mk}},$$
with $\rho_{mm}=1$. This is the standard design-effect formula for correlated samples. It equals the plain weighted count when errors are independent and collapses toward a single vote when the extractors are duplicates.
The effective counts become a Subjective Logic opinion. Belief in label $c$ is $b_c = n_c / (\sum_{c'} n_{c'} + W)$ and uncertainty is $u = W / (\sum_{c'} n_{c'} + W)$, where $W$ is a fixed prior weight (we use $W=2$, the usual binary-opinion convention). The selection score is $E_c = b_c + a_c u$, where $a_c$ is the base rate of relation type $c$. Confident disagreement is handled as in Discounted Belief Fusion, with one difference: the discount threshold is chosen on calibration data by maximizing coverage at the target risk. This addresses the tuning sensitivity that the file reports for that method.
Finally, the score is passed to two wrappers. One applies the per-class binomial risk bound of Selection with Guaranteed Risk. The other applies Act-or-Defer-style stratified conformal selection, which uses Benjamini–Hochberg control and treats each interaction subtype (Mechanism, Effect, Advice, Int) as its own stratum with its own error budget. Accepted facts are committed to the database, and the remainder forms the curator queue.
There is one secondary analysis. Because $u$ is kept separate from conflict, we can mark a deferred item either as "sparse" (few extractors fired, little evidence) or as "contested" (strong evidence pointing in opposite directions). The file raises this distinction for under-studied entities in its discussion of Affinage. We test whether the two flags predict different curator outcomes. We report this as diagnostic, not as a headline claim.
Baselines
The naive reference point commits everything. On DDI'13, the strongest forced-choice result in the file is the 10-model Tree-LSTM ensemble of Lim et al., at 83.8% detection F1 and 72.7% type-classification F1. The majority-voting system WBI-DDI reports 73–76% detection F1 and 60.9% subtype F1. These numbers set the error rate a curator would inherit with no abstention. We found no reported result on Lacuna for selective risk–coverage on DDI'13, so every abstaining baseline below is run by us.
All abstaining methods share the same extractor outputs, the same calibration/test splits (100 random resplits), the same wrapper, and the same per-class targets. Only the score changes. Any coverage difference is therefore due to ranking quality, and the risk guarantee holds equally for all of them.
The first abstaining baseline is the best single model's confidence (softmax response or normalized token likelihood) under Selection with Guaranteed Risk. That paper found this simple score competitive with Monte Carlo dropout, which makes it the minimal serious competitor.
The second is self-consistency: repeated sampling of the best open model, scored by answer agreement, matched to the ensemble's inference budget.
The third is raw vote fraction across the full pool, and the fourth is vote fraction weighted by the Optimal Weight rule of Beyond Majority Voting. The file reports that rule as 3.36% above majority voting on MMLU and more than 7% above it on contested questions, in forced choice. We consider weighted vote fraction the strongest existing competitor, because it already uses accuracy and correlation information.
The fifth is ensemble entropy deferral, as in CollabGeneral.
The sixth is Discounted Belief Fusion without the correlation discount. This serves as the key ablation: it isolates the effective-count term from the rest of the evidential machinery.
The seventh is a logistic-regression stacker trained on all extractor confidences using the same calibration labels.
Beating a baseline means higher coverage at equal realized per-class risk, averaged over resplits. The pre-registered threshold is a gain of at least five coverage points over weighted vote fraction at $r^* = 5\%$ on the Int and Advice strata, with no increase in the rate of guarantee violations.
Why not just…?
Why not just threshold one strong model's confidence? This is a genuine threat. Affinage reached 97.95% literature support with a single pipeline. However, it has no risk guarantee, and both it and the comparative molecular-interaction study report degradation on under-studied entities. On short-form QA, ASPIRE shows that raw likelihoods are poorly calibrated, with AUROC on CoQA rising from 74.61% to 80.25% only after learned self-evaluation. We run this baseline under the identical wrapper. If it matches our method's coverage, we will conclude that ensembles are unnecessary for this task.
Why not just use vote fraction, or accuracy-weighted vote fraction? Our predicted failure mode is concrete. Duplicating a mediocre extractor inflates its share of the vote, so vote fraction promotes exactly the facts on which the duplicated lineage is wrong. The effective count is nearly invariant to such duplication. The duplication experiment below tests this directly. If weighted vote fraction remains as good as our method once correlated copies are added, the correlation term is not earning its place.
Why not just train a stacker on the calibration labels? A stacker can learn correlations implicitly, and with ample labels it may match or beat a fixed formula. Our bet is about label efficiency. The effective-count score has roughly one parameter per extractor pair, whereas a stacker must learn interactions from scarce expert labels. We therefore sweep calibration-set size from 200 to the full split. If the stacker wins at every budget, we report that result plainly, and the contribution narrows to the correlation analysis.
Why not just use self-consistency sampling? Diversity is the strength of the AI crowd suggests that repeated samples from one model inherit its blind spots, and ASPIRE notes that sampling multiplies inference cost by the number of samples. Matching inference budgets lets us measure both effects rather than assume them.
Proposed Experiments
The primary benchmark is DDI'13, where the Int and Advice subtypes provide naturally rare strata. A second benchmark is binary protein–protein interaction verification against STRING positives and Negatome 2.0 negatives, from the comparative LLM evaluation. That study reports no exact F1 values; we found no reported result on Lacuna for selective risk on that suite. As a qualitative stress test of the sparse-versus-contested flags, we use the Affinage 100-gene failure-enriched cohort, which includes 50 hard cases with alias collisions or no known mechanism.
The metrics are coverage at fixed per-class risk, area under the risk–coverage curve, empirical guarantee-violation frequency across resplits, and curator workload (the fraction of items deferred) at matched database error.
Three checks decide the outcome. The first is a duplication test: we add $k = 0$ to $4$ near-copies of one extractor and plot coverage at $r^*=5\%$. Our method should stay roughly flat while vote fraction degrades. The second is a dose–response test: across random sub-ensembles, the coverage gain over vote fraction should increase with the ensemble's mean pairwise error correlation and should approach zero for low-correlation ensembles. If the gain does not track correlation, the mechanism is wrong even if the average gain is positive. The third is a rare-class test: under stratified selection, our score should commit more correct Int and Advice relations than any baseline while keeping violation rates at or below $\delta$. On those strata, scores computed from correlated majorities are the ones most likely to overrank shared errors.
References
- Selective Classification for Deep Neural Networks
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Quantifying Correlations of Machine Learning Models
- Diversity is the strength of the AI crowd
- Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information
- Multimodal Learning with Uncertainty Quantification based on Discounted Belief Fusion
- Drug drug interaction extraction from the literature using a recursive neural network
- Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge
- Affinage: Genome-Scale Mechanistic Gene Annotation from the Published Literature
- WBI-DDI: Drug-Drug Interaction Extraction using Majority Voting
- On Optimizing Model Generality in AI-based Disaster Damage Assessment: A Subjective Logic-driven Crowd-AI Hybrid Learning Approach
- Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs