When biologists hunt for molecules that bind a disease target, they often use phage display: a library of billions of random protein variants is physically mixed with the target, weak binders are washed away, and the survivors are copied and put through the cycle again. With cheap next-generation sequencing, each round now yields read counts for millions of variants, and the field increasingly ranks candidates by how strongly their counts grow. The problem, established empirically, is that these counts are treacherous: fast-replicating "cheater" viruses and sticky variants that glue themselves to plastic beads or blocking proteins can dominate the readouts while having nothing to do with the target, and sampling noise alone scrambles rankings (Bayes-PD: Exploring a Sequence to Binding Bayesian Neural Network model trained on Phage Display data; SAROTUP: a suite of tools for finding potential target-unrelated peptides from phage display data).
This proposal imports a mature measurement theory from a completely different field — inter-annotator agreement, the statistics used to decide whether human labelers genuinely agree or are agreeing by accident. The core move is to treat each screening experiment (a replicate panning run, a counter-selection against a plastic-only well, a screen against an unrelated target) as an annotator that casts a noisy vote on every sequence: enriched or not. Two decades of agreement research tell us exactly how to score such votes, and exactly how the naive versions fail under the extreme class imbalance that defines phage display, where true binders are rarer than one in a million. No published work has ever applied chance-corrected agreement statistics to biopanning data; the closest efforts stop at probabilistic count modeling on one hand and uncalibrated heuristic rules on the other.
Core Claim
If we score cross-screen concordance with prevalence-independent agreement statistics — and pair cross-screen agreement with within-screen replicate stability, borrowing the reliability–stability logic of Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement — we can statistically separate three populations that current pipelines confound: genuine target-specific binders, target-unrelated artifacts (plastic-, streptavidin-, or propagation-driven false positives), and stochastic wash-through noise. The same per-sequence concordance scores can then serve as sample weights or a training curriculum for sequence-to-affinity models, addressing a problem that Bayes-PD explicitly leaves open: disentangling positive selection signal from negative selection and propagation bias.
Two measurement facts make this more than a relabeling exercise. First, raw overlap counting cannot distinguish signal from accident: the chance-corrected coefficient
$$\kappa = \frac{P_o - P_e}{1 - P_e},$$
where $P_o$ is the observed fraction of screens agreeing on a sequence's enrichment status and $P_e$ is the agreement expected from each screen's marginal call rates, exists precisely because two noisy screens can agree voluminously by chance. Second, and more severely, The Problem with Kappa proves that Cohen's and Fleiss' versions of this coefficient collapse toward zero or turn negative under prevalence skew — and phage display is the most skewed regime imaginable. The proposal therefore builds its concordance score on Powers' prevalence-independent reformulation, Informedness,
$$\text{Informedness} = \text{sensitivity} + \text{specificity} - 1,$$
computed per sequence against a pooled consensus reference, and on Krippendorff's alpha, which tolerates the missing-data pattern that arises because no screen observes every sequence.
Research Program
The first deliverable is a formal mapping from screening anatomy to agreement-measurement structure. Technical replicates — the same library panned against the same target in parallel — play the role of intra-annotator stability (one judge re-labeling the same items). Screens against different targets, or counter-selection runs against bare plastic, streptavidin beads, and blocking proteins, play the role of inter-annotator reliability (different judges on shared items). Crossing these two axes yields a clone-level taxonomy that the reliability–stability matrix makes precise: sequences with high within-target stability but low cross-target concordance are candidate specific binders; sequences stable and concordant across unrelated targets are target-unrelated artifacts of the selection or propagation machinery; unstable, non-concordant sequences are wash-through noise. This taxonomy is exactly what SAROTUP's binary filters and library-locked SVMs approximate with heuristics, and what Bayes-PD cannot produce because it models rounds within one experiment as a single stochastic process rather than as independent judges.
The second deliverable is a chance-corrected, per-sequence enrichment score. For each sequence we collect its binarized enrichment calls across all available screens (with thresholds set per screen by its own null distribution, so margs are comparable), compute the sequence's contribution to pooled Informedness and alpha, and attach an uncertainty via the count noise model that Bayes-PD already validated — a Poisson approximation to multinomial read counts. The output is a calibrated probability that a sequence's enrichment is target-specific, replacing the raw fold-change ranking.
The third deliverable uses these scores downstream. Following Learning part-of-speech taggers with inter-annotator agreement loss, which weighted training loss by how much human labelers agreed on each token, we weight the training instances of an ESM-2-embedded Bayesian neural network (the Bayes-PD architecture) by their concordance score. Following Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification, which staged training from unanimous pathologist labels down to disputed ones, we train first on high-concordance sequences and introduce low-concordance ones gradually. A genuine open question rides along here: the histopathology study found a ceiling — its most disputed tier (4-of-7 agreement) hurt performance, dropping the gains — while the POS-tagger study found confusion-weighted loss helped universally. Whether borderline phage clones behave like pathological edge cases (informative) or like Stage-4 noise (destructive) is untested in any molecular screening context, and our experiments are designed to answer it, not assume it.
Proposed Experiments
Three experiments, each anchored to a benchmark in the file. The first uses the Biopanning Data Bank, which archives 3,291 biopanning datasets from 1,540 articles (95 with next-generation sequencing) across 1,971 targets. We assemble, for every peptide observed in screens against two or more unrelated targets, the multi-screen call matrix, and test whether our concordance score recovers known target-unrelated peptides better than the heuristic alternatives. Ground truth comes from two sources: SAROTUP's own flags, and an independent negative set built from the naive Ph.D.-7 deep-sequencing dataset described in SAROTUP — 3.05 million unique 7-mer sequences from an unpanned library, in which 36,197 were flagged as TUPs by motif scanning — since a variant's presence and behavior before any selection calibrates its null distribution. Metrics are precision, recall, and Informedness of TUP detection, plus calibration of the specificity probability against held-out targets.
The second experiment uses the Bayes-PD benchmark: multi-round, multi-concentration count data for three protein targets with biophysically measured dissociation constants ($K_d$). We retrain the Bayes-PD network under concordance-weighted loss and under the concordance curriculum, holding the architecture, embeddings, and Poisson likelihood fixed so any change is attributable to the agreement signal alone. The decisive number is correlation with measured $K_d$ on the held-out BFN-designed sequences, where Bayes-PD as published reports $r = -0.46$ (Bayes-PD); beating that figure with the same base model would be direct evidence that agreement weighting extracts affinity signal that uniform weighting misses. No reported result found on Lacuna for a competing concordance-weighted model on this dataset, so this comparison defines the frontier.
The third experiment is a controlled spike-in: we construct synthetic screen ensembles (by subsampling the naive-library reads and planting known TUP and binder populations with prescribed enrichment dynamics) where the ground-truth quadrant of every sequence is known by construction. This is where the reliability–stability decomposition is stress-tested — specifically whether propagation-driven cheaters (enriched regardless of target) can be separated from plastic-binders (enriched on matrix-heavy but not counter-selected screens), which no current tool distinguishes, and whether replicate count (three versus five versus ten screens) shifts the operating point as the statistics predict.
Baselines
The naive reference point is raw round-over-round fold enrichment: rank sequences by their read-count growth between selection rounds and take the top $k$. This is what most practitioners actually do, and it is the correct null hypothesis to beat; Bayes-PD found such raw selectivity measures correlate poorly with $K_d$, which is why beating them should be easy on affinity prediction but must still be verified on TUP recovery, where no such test has been run. The strongest existing competitor for TUP detection is the SAROTUP 3.1 suite (SAROTUP): motif scanning against curated TUP dictionaries, BLAST-style cross-referencing against prior screens in the Biopanning Data Bank, its "Four-Target Rule" flagging any peptide enriched against four or more unrelated targets, and its library-specific SVMs for propagation advantage and plastic/streptavidin binding. We will run SAROTUP on exactly our assembled multi-screen datasets with the same input sequences and the same compute (its authors report sub-hour runtimes on a laptop-class machine, so compute is not a constraint), and score both methods against the same ground-truth labels. The file reports SAROTUP's runtime and its isolation of 36,197 TUPs from 3.05 million naive-library sequences, but no reported result found on Lacuna for its TUP-detection precision or recall — so our spike-in experiment, where truth is known, is what makes this comparison meaningful. Beating SAROTUP means strictly higher TUP-recovery Informedness at matched false-positive rate on held-out targets, and the ability to score libraries SAROTUP cannot (Ph.D.-12, custom nanobody libraries), since its propagation predictor is calibrated only for Ph.D.-7. The strongest existing method on the affinity side is published Bayes-PD itself at $r = -0.46$ with $K_d$; we beat it only if concordance weighting of the identical architecture and data exceeds that correlation with confidence intervals that exclude it. The key ablation replaces the chance-corrected score with an uncorrected co-occurrence rate (the raw fraction of screens in which the sequence was enriched) while keeping everything else — same stability axis, same weighting scheme, same model — so that any gain is attributable to chance correction specifically. A second ablation drops the stability axis (using cross-screen agreement only), isolating the contribution of the reliability–stability pairing. A third trains on the full curriculum including the lowest-concordance tier, versus stopping at the middle tier, to measure where the disagreement ceiling sits in this domain.
Why not just…?
Why not just threshold on read counts or fold change? Because the two failure modes this proposal targets are invisible to fold change. SAROTUP documents that propagation-advantaged clones exponentially dominate late-round counts regardless of binding, and Bayes-PD shows raw selectivity ratios are corrupted by sampling bottlenecks and stochastic amplification, requiring full probabilistic noise modeling to reach even $r = -0.46$ with $K_d$. Fold change cannot distinguish a cheater from a binder because both look enriched; only cross-target concordance separates them, and only chance correction keeps that separation honest.
Why not just take the intersection of replicate screens (majority vote)? Majority voting is the uncalibrated ancestor of everything proposed here, and it fails in a documented way: it cannot distinguish stable artifacts from genuine consensus, because a plastic-binding clone appears in every replicate too. Consistency is Key shows that majority votes across judges who are individually inconsistent (humans self-contradict about 25% of the time on re-labeling) pool noise rather than resolve it, and that only the joint reliability–stability view separates stable-but-wrong judges from noisy ones. Translated to screens: intersection of same-target replicates filters wash-through noise but retains every selection-related artifact; adding the cross-target axis — and correcting for chance overlap — is precisely the ingredient majority voting lacks. Our ablation against uncorrected co-occurrence quantifies exactly this gap.
Why not just use SAROTUP as it stands? For the Ph.D.-7 peptide library with previously deposited screens, SAROTUP is a real threat, and we treat it as the primary baseline rather than argue around it. Its own limitations define where it must lose: its propagation predictor is locked to Ph.D.-7, it has no predictors for bovine serum albumin, protein A/G, or novel assay components, and its Four-Target Rule requires the artifact to already appear in the historical database — blind to anything new in custom screening setups (SAROTUP). Our spike-in experiment plants artifacts that are absent from BDB precisely to measure this blind spot; if SAROTUP matches our recovery there, the proposal's filtering contribution is genuinely in question, and we would say so.
Why not just compute Cohen's or Fleiss' Kappa on the binarized screens? Because the setting is the pathological corner case those statistics were shown to fail in. The Problem with Kappa demonstrates that under extreme prevalence skew — and binders rarer than one in $10^6$ is more extreme than any of Powers' simulations — expected chance agreement approaches one, the denominator $1 - P_e$ collapses, and Kappa drops below zero even for accurate raters. This is why the proposal builds on Informedness and Krippendorff's alpha from the outset rather than retrofitting standard Kappa and discovering the paradox empirically.
Why not skip the messy screening data and design binders entirely in silico? Structure-guided generative pipelines such as Structure-Guided Reinforcement Learning for High-Affinity Antibody Design and the distilled predictor of A Simple yet Effective $\Delta\Delta G$ Predictor is An Unsupervised Antibody Optimizer and Explainer optimize designs against computable energy models, but both acknowledge that their outputs require wet-lab validation, and physical screening remains the source of ground truth. This proposal is orthogonal: it makes the wet-lab screening data itself more trustworthy, which is valuable exactly because in silico pipelines ultimately need such data to validate and retrain against.
Decisive Checks
The proposal stands or falls on three numbers. On the spike-in benchmark, concordance-based TUP recovery must beat both the fold-change ranking and SAROTUP 3.1 in prevalence-independent Informedness at matched false-positive rate, including on planted artifacts absent from public databases. On the Bayes-PD benchmark, concordance-weighted training of an otherwise unchanged model must push the correlation with measured $K_d$ beyond the published $r = -0.46$. And the ablation against uncorrected co-occurrence must show a measurable gap, because if chance correction adds nothing over raw overlap counting, the agreement-theoretic machinery — the whole intellectual import of the intersection — would be decorative rather than load-bearing, and the honest conclusion would be that simple multi-screen intersection suffices.
References
- Bayes-PD: Exploring a Sequence to Binding Bayesian Neural Network model trained on Phage Display data
- SAROTUP: a suite of tools for finding potential target-unrelated peptides from phage display data
- Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
- The Problem with Kappa
- Learning part-of-speech taggers with inter-annotator agreement loss
- Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification
- Biopanning data bank 2018: hugging next generation phage display
- Structure-Guided Reinforcement Learning for High-Affinity Antibody Design
- A Simple yet Effective $\Delta\Delta G$ Predictor is An Unsupervised Antibody Optimizer and Explainer