Measuring Real Agreement Between Peptide Search Engines at the Detection Limit

When a protein is present at only a trace amount, the mass spectrometer records weak, gappy fragment spectra. Mass spectrometry here means breaking peptides into charged fragments and reading their mass-to-charge ratios. Proteomics labs therefore often run several identification programs ("search engines") and trust a peptide when the programs agree. This note argues that such agreement is poorly understood exactly where it matters most. On a sparse spectrum, two engines can pick the same wrong peptide because they share a candidate list narrowed by mass tolerance, the same scoring biases, or the same amino-acid priors. Annotation research learned long ago that raw agreement is misleading whenever agreement by coincidence is likely. Its fix is chance-corrected agreement, which subtracts the agreement you would expect if the raters knew nothing.

We propose to import that idea in a form that suits mass spectrometry. Standard agreement coefficients estimate chance agreement from the raters' label frequencies. Over a label space of millions of peptides, that estimate is meaningless. Instead, we measure chance agreement empirically. We search spectra against databases from which the true peptide is guaranteed to be absent, and we count how often the engines still converge on the same answer, stratified by signal intensity and spectral sparsity. The resulting per-intensity curve of "agreement beyond chance" does two jobs. It is a diagnostic, telling us whether multi-engine consensus carries any information in the lowest-abundance regime. It is also a calibration device for accepting consensus identifications with an error rate that accounts for correlated mistakes.

Core Claim

The claim is that agreement between peptide identification engines is strongly correlated on low-intensity spectra. Naive consensus and per-engine false discovery rates therefore misstate the error rate of consensus calls. A false discovery rate (FDR) is the expected fraction of accepted identifications that are wrong.

For each intensity or sparsity bin $b$, write

$$\kappa_b = \frac{P_{o,b} - P_{e,b}}{1 - P_{e,b}}.$$

Here $P_{o,b}$ is the observed fraction of spectra in bin $b$ on which two engines return the same peptide. $P_{e,b}$ is the fraction on which they return the same peptide when the correct sequence has been removed from the search space. Both quantities are measured, not modeled. We expect $\kappa_b$ to fall much faster than raw agreement as intensity drops.

The same contrast should explain part of the "noise paradox" reported by NovoBench. NovoBench found that de novo models do worse on low-noise spectra, because those spectra tend to have extreme missing fragmentation. In that regime we predict raw agreement among de novo models stays high, since they share canonical-sequence priors, while $\kappa_b$ approaches zero. The models would be agreeing the way inconsistent survey respondents agree on "non-attitudes" in Position: RLHF May Not Reflect Genuine Preferences: their agreement is shared default behavior rather than shared evidence.

Relation to Prior Work

The annotation literature supplies the method and the motivating warnings.

The general coefficient above is the common core of Cohen's kappa, Fleiss' kappa and Krippendorff's alpha. Each estimates $P_e$ from label marginals, and our key departure is to replace that with an empirical null. Three papers motivate the move:

  • Handling Inter-Annotator Agreement for Automated Skin Lesion Segmentation measured a median kappa of about 0.72 among dermatologists. It argued that a model claiming far higher agreement with one rater is memorizing that rater's quirks. We borrow its logic of an agreement ceiling: consensus cannot be more trustworthy than its chance-corrected agreement allows.
  • That same paper showed that crude geometric consensus, such as bounding boxes, made agreement worse. This is a warning against heuristic reconciliation, and it applies directly to Venn-diagram intersection of engine outputs.
  • Learning part-of-speech taggers with inter-annotator agreement loss weighted training errors by human confusion rates. The resulting gains were modest but real out of domain, for example downstream chunking F1 rose from 68.49 to 68.97. We use its insight that the structure of disagreement is informative. We apply it to evaluation and acceptance rather than to training loss.

The proteomics literature supplies the raters and constrains the design.

  • A learned score function improves the power of mass spectrometry database search introduces Casanovo-DB, a transformer-derived database scorer. At 1% FDR it reports 31–40% more peptides than XCorr, 52–69% more than Hyperscore, and 88–102% more than Andromeda. That makes it the single engine to beat, and it supplies four engines with very different scoring philosophies.
  • DIAmeter handles data-independent acquisition. That acquisition mode fragments all peptides in a mass window at once, so fragment spectra from different peptides overlap. DIAmeter matches these spectra to database peptides using several orthogonal features and Percolator rescoring. Percolator is a learned classifier trained against decoy, i.e. deliberately fake, peptides. DIAmeter shows that combining features is standard practice. It never asks whether the feature extractors, or separate engines, make correlated errors on weak signals.
  • AdaNovo down-weights uninformative spectra using a mutual-information score computed inside a single model. Our signal comes instead from agreement across independent systems.
  • LIPNovo, with code at usr922/LIPNovo, reports that its imputation of missing fragments breaks down above roughly 70–80% missing fragmentation. That regime is where we expect agreement to be least informative, so it marks the region our diagnostic must cover.

None of these papers measures chance-corrected agreement across engines, and none estimates chance agreement from absent-truth searches.

Research Program

The program has three parts.

The agreement null. For each benchmark we build entrapment searches. We append sequences from proteomes known to be absent from the sample, and in separate runs we remove each spectrum's confidently identified peptide from the database. We then rerun every engine. Engine pairs and the full panel are scored for exact sequence agreement, with isoleucine and leucine treated as identical. We also score a relaxed agreement that ignores modification localization.

Stratified agreement curves. Spectra are binned by precursor intensity and by the missing-fragmentation ratio and noise-signal ratio that NovoBench defines. In each bin we compute observed agreement, $P_{e,b}$, and $\kappa_b$, with bootstrap intervals over spectra.

Consensus acceptance under correlated errors. We define the consensus acceptance rule: accept a peptide in bin $b$ when $k$ of $m$ engines agree and each passes a per-engine score threshold. The FDR of that rule is estimated from the frequency of agreeing calls in the entrapment runs. Its thresholds are chosen per bin to hold this estimate at 1%. This uses agreement directly as evidence with a measured null, rather than as an untested assumption.

Baselines

Single best engine. The do-nothing reference is the best single engine with its standard error control: Casanovo-DB followed by Percolator, as in A learned score function improves the power of mass spectrometry database search. There, Percolator adds about 9% more peptides on average and Casanovo-DB leads XCorr by 31–40% at 1% FDR on the post-2019 three-organism benchmark. We rerun it on exactly the same spectra and databases as our method, using the published settings. Beating it means more distinct peptides in the lowest-intensity tertile while matching its actual false discovery proportion, i.e. the realized fraction of wrong calls. We measure that proportion independently through entrapment hits.

Naive intersection. The engine panel is unchanged and a peptide is accepted when all engines report it at their own 1% FDR. This is what many labs do. We expect its estimated error rate to be inaccurate, in either direction, in the low-intensity bins.

Multi-engine Percolator. All engines' scores are concatenated into a single Percolator model with joint target-decoy competition. This is the strongest existing approach, and the most dangerous competitor, because Percolator can in principle learn agreement patterns on its own. It receives the same features, the same decoys and the same cross-validation folds as our method.

Ablations. The key ablation isolates our main ingredient by replacing the entrapment-measured $P_{e,b}$ with Cohen's marginal-based $P_e$. A second ablation removes stratification and uses a single global $\kappa$.

For data-independent acquisition we use DIAmeter and Dear-DIA as the two raters. There, "no reported result found on Lacuna" for exact DIAmeter identification counts on LFQBench. The only quantitative anchor in the file is Dear-DIA's report of 12.3 times more unique peptides than DIA-Umpire on HYE124, from Dear-DIA XMBD.

Why not just…?

Why not just rely on target-decoy competition and Percolator, which already control error? Target-decoy competition controls a global error rate for one scoring scheme under the assumption that decoys behave like false targets. It says nothing about whether two engines' false targets coincide. Aggregate statistics can also hide strong local inconsistency, as the RLHF diagnostics paper shows for human ratings. Our entrapment-measured false discovery proportion, computed per intensity bin, directly tests whether Percolator-based estimates stay accurate in the lowest bin. If they do, that is a useful negative result and we report it.

Why not just put every engine's score into one Percolator model? This is a real threat. If multi-engine Percolator matches our method's yield at equal measured false discovery proportion, then our acceptance rule adds nothing as a detector. The contribution would narrow to the diagnostic: curves showing where agreement stops being evidence, which Percolator does not expose. We run exactly this comparison. One further experiment is planned: adding $\kappa_b$ as a Percolator feature, to see whether the measured null helps even inside the learned combiner.

Why not just take the intersection of engines? Intersection assumes that the engines' errors are independent. The skin-lesion study shows that crude consensus can destroy information, and the shared mass-tolerance candidate pools make coincidental agreement plausible. The entrapment runs measure this coincidence directly: if $P_{e,b}$ is negligible in every bin, intersection is vindicated.

Why not just reweight training with internal information scores, as AdaNovo does? AdaNovo addresses a different question, namely which training spectra to trust inside one model, and it adds about 40% more parameters. LIPNovo later outperformed it, which suggests that internal reweighting cannot recover missing evidence. Our measure needs no retraining and evaluates agreement across models. We include AdaNovo as one of the de novo raters rather than as a rival.

Proposed Experiments

Database-search experiment. The main experiment uses the post-2019 three-organism benchmark with XCorr, Hyperscore, Andromeda and Casanovo-DB.

De novo experiment. We run Casanovo, AdaNovo and LIPNovo on the nine-species and HC-PT benchmarks. In HC-PT the synthetic peptides give true sequences, so true agreement can be checked directly.

DIA experiment. We use LFQBench and HYE124 for data-independent acquisition. Their known species composition provides built-in entrapment, and their mixture ratios place some species at low abundance. The plasma dataset serves as a stress test with a very wide range of protein abundances.

Metrics. We report the following:

  • distinct peptides at a measured 1% false discovery proportion, per intensity tertile;
  • calibration error between estimated and measured false discovery proportion;
  • the $\kappa_b$ curves themselves.

Decisive checks. Three outcomes decide the main questions:

  • Whether agreement is correlated. The core claim fails if entrapment-measured $P_{e,b}$ stays near zero in every bin, meaning engine errors are effectively independent.
  • Whether the noise-paradox prediction holds. In NovoBench's low-noise, high-missingness bins, de novo raw agreement should remain high while $\kappa_b$ collapses.
  • Whether the consensus rule is worth adopting. In the lowest-intensity tertile at equal measured error, our rule must beat both Casanovo-DB with Percolator and multi-engine Percolator. If only the first two outcomes hold, the work stands as a reliability diagnostic for trace-level proteomics rather than a new detector.