When a retrieval system needs precision, it almost always adds a second stage: a large model looks over the top candidates from a cheap first-pass search and re-orders them. That re-ranking stage is where the quality comes from, and also where the cost lives — a large language model scoring candidates one at a time needs a full forward pass per candidate, and comparing candidates side by side inflates the prompt until attention costs blow up. Nowhere is this trade-off sharper than in the antibiotic resistance crisis, where researchers and clinicians routinely need to rank thousands of candidate antimicrobial peptides or pieces of clinical evidence, and where a wrong ranking wastes wet-lab effort or, worse, steers therapy toward drugs a pathogen already resists. There is a second, quieter problem: large-model rankings are brittle. Rewrite the query slightly, shuffle a list, or add a polite word, and the ranking can flip — a fragility documented across leaderboards and medical question answering alike.
This proposal argues the two problems share one solution in a specialized setting. We will build a compact (at most ~3 billion parameter), domain-specialized re-ranker for antimicrobial candidate retrieval that emits, in a single forward pass, both a ranking and a per-candidate confidence score — the confidence derived from how much the model's own internal scoring machinery disagrees with itself, not from extra inference calls. The model will be trained with reinforcement learning whose rewards are checked by a deterministic peptide-calculator (net charge, hydrophobicity, molecular mass), which both grounds the ranking in biophysics and, if the training works as expected, suppresses the phrasing sensitivity that plagues prompted models. In a clinical antibiotic-stewardship setting, low-confidence rankings route to a human pharmacist instead of silently proceeding.
Core Claim
The claim has three parts, and each is falsifiable separately. First, a compact re-ranker specialized on antimicrobial sequence and literature data can match or beat both first-stage retrieval and general-purpose LLM re-rankers on antimicrobial retrieval tasks, at a fraction of the inference cost. Second, the dispersion of relevance scores across attention heads inside one forward pass is a usable proxy for the expensive kind of sensitivity measurement — running the same query under many paraphrased prompts and measuring output variance — so the system knows when it is unstable without paying the multi-pass price. Third, reinforcement learning with a programmatic biophysical verifier reduces the re-ranker's own prompt sensitivity, where supervised fine-tuning alone does not.
No published work in the research file does all three, or even any two together in this setting. The closest pieces: ASRank (ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval) solves the cost problem with a two-tier design — one large-model call produces a short hypothetical "answer scent," and a small T5 model scores candidates against it — but it produces no confidence signal, never varies its prompts, and was evaluated only on open-domain question answering. SENSEL (On the Relation between Sensitivity and Accuracy in In-Context Learning) turns prompt-stability into a confidence signal, but needs one forward pass per prompt variant, which is exactly the overhead this proposal refuses to pay, and it was tested only on text classification. AttriLens-Mol (AttriLens-Mol: Attribute Guided Reinforcement Learning for Molecular Property Prediction with Large Language Models) shows verifier-grounded RL can stabilize a small model on molecular tasks, but it predicts molecular properties; it never ranks retrieval candidates. RankNovo (Universal Biological Sequence Reranking for Improved De Novo Peptide Sequencing) re-ranks biological sequences, but for mass-spectrometry peptide sequencing, with latency dominated by running multiple base models, no natural-language queries, and no confidence output. Our proposal is the composition: ASRank's amortized architecture, SENSEL's sensitivity-as-confidence idea made cheap, and AttriLens-Mol's verifier-grounded training, all pointed at antimicrobial retrieval.
Research Program
The system has three components. The first is the candidate generator and first-stage retriever: a bi-encoder that embeds peptide sequences and query text independently and returns the top candidates by cosine similarity. For peptides we build on the dual-branch transformer from the ESCAPE benchmark (A Standardized Benchmark for Multilabel Antimicrobial Peptide Classification), which fuses sequence encodings with predicted contact-distance maps; for clinical-literature retrieval we use a standard dense retriever. The first stage supplies recall; it cannot supply precision, which is the known division of labor in retrieval.
The second component is the re-ranker itself, an encoder-decoder model of roughly 220M to 3B parameters using Fusion-in-Decoder, a design where each candidate is encoded separately and the decoder attends over all of them, keeping cost linear in candidate count. LiT5 (LiT and Lean: Distilling Listwise Rerankers Into Encoder-Decoder Models) showed that a relevance score can be read directly out of the decoder's cross-attention weights without re-ranking-specific labels, and that a 220M model of this shape can beat a 7B listwise re-ranker. We transplant ASRank's query-side trick into the biological domain: one call to a large model converts a natural-language query such as "peptides active against carbapenem-resistant Gram-negatives with low hemolysis" into a structured attribute scent — target function class, net charge range, hydrophobicity window, length, mass. The compact re-ranker then scores every candidate against that scent. This is one large-model call per query regardless of candidate count, the $O(1)$ property that makes ASRank roughly 42% faster than its closest competitor and 97% cheaper than prompted LLM re-ranking.
The third component is the confidence signal, and it is the main technical bet. Instead of running $K$ prompt variants, we measure disagreement inside the single pass. Let $s_i^{(h)}$ be the cross-attention-derived relevance score for candidate $i$ under attention head $h$; the confidence is
$$c_i \;=\; 1 - \mathrm{std}_{h}\!\left(s_i^{(h)}\right),$$
normalized so $c_i$ near 1 means the model's heads agree the candidate is (or is not) relevant. The intuition comes from ProSA (ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs): when a model lacks solid grounding, its output distribution is diffuse, and small input changes push it across decision boundaries. We hypothesize that head-level disagreement is a visible symptom of the same diffuse internal state, and that it correlates with true multi-prompt sensitivity. We calibrate $c_i$ against the expensive ground truth — 16-prompt paraphrase variance, computed offline at evaluation time only — and use the calibrated value for selective abstention.
Training proceeds in two phases: supervised fine-tuning on teacher-labeled ranking pairs, then reinforcement learning with Group Relative Policy Optimization in the style of AttriLens-Mol, but with a peptide-appropriate verifier. AttriLens-Mol's rewards were limited to 2D topological properties computable by RDKit, which cannot verify peptide biophysics; our verifier instead computes net charge, GRAVY hydrophobicity, and molecular mass deterministically, in the spirit of RankNovo's mass-deviation losses, and rewards the model when the attributes it cites for a candidate match the verifier's computation. A count penalty discourages long rationales, so the model does not spend tokens explaining itself at inference time — a deliberate rejection of InteRank's approach (Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking), which raises accuracy by generating full explanations but pays that accuracy back in latency, and whose reward model is ungrounded, causing regressions on simpler tasks.
Baselines
The naive reference point is the first-stage retriever alone, with no re-ranking. This baseline matters because it defines what the re-ranker must add, and the literature quantifies the gap: in ASRank's experiments on Natural Questions, first-stage retrieval reached only 19.2% top-1 accuracy, which re-ranking lifted to 46.5%; on the reasoning-heavy BRIGHT benchmark, dense retrievers such as BGE and E5 scored below 20% nDCG@10 while InteRank's re-ranker reached 27.4%; and on BEIR's COVID split, HyQE (HyQE: Ranking Contexts with Hypothetical Query Embeddings) lifted Contriever's 27.32 NDCG@10 to 35.54. We will run our first-stage encoders unmodified on every task with identical candidate pools, and report the same metrics; beating it is necessary but not interesting.
The strongest existing competitor is ASRank itself, because it is the published system closest to our architecture: one $O(1)$ large-model scent call plus a compact T5 cross-attention scorer, reporting 46.5% top-1 on Natural Questions (against a 35.4% unsupervised-passage-reranking baseline), a 42% latency reduction, and a 97% evaluation-cost reduction versus prompted LLM re-ranking. We will reimplement ASRank faithfully, changing only the scent generator's domain prompt so it produces attribute scents for our queries, and run it on exactly the same candidate sets, the same first-stage retriever, and the same hardware, with the same tuning budget (identical hyperparameter search ranges and training epochs) granted to our model. Beating ASRank means two things jointly: equal or better ranking quality (mAP or nDCG@10) at equal or lower wall-clock latency, and a calibrated confidence signal, which ASRank does not produce at all — so on the confidence axis any validated signal wins, but on ranking quality we must not lose, or the proposal fails.
The third baseline is InteRank's recipe — supervised distillation from a 70B teacher plus RL refinement with an ungrounded reward model — applied to our data, since it holds the best reported number on reasoning-intensive retrieval (27.4% nDCG@10 on BRIGHT). The fair comparison is at matched total inference FLOPs: InteRank's explanations consume tokens per candidate, so we compare quality per unit of inference compute, not per model. Finally, two ablations isolate the proposal's key ingredients: the identical architecture trained with supervised fine-tuning only (no verifier RL), which isolates the RL contribution, and confidence computed from maximum softmax probability and entropy instead of head dispersion, which isolates the confidence mechanism. For confidence quality, SENSEL's K-pass method — reported at +4.1 AUC over MaxProb on the F1-coverage curve for GPT-J-6B — serves as the expensive upper bound.
Why not just…?
Why not just run the query under K paraphrased prompts and take a majority vote? This is the most serious alternative, and it is the accuracy gold standard for stability: SENSEL shows stability across prompt perturbations beats MaxProb and entropy by +4.1 AUC as an abstention signal, and the sensitivity-accuracy correlation it exploits reaches -0.40. But it multiplies inference cost by exactly K, the overhead this whole line of research exists to eliminate, and in the antimicrobial setting the candidate sets are large enough that K-pass scoring is prohibitive; moreover, Rethinking Hallucinations (Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity) shows retrieval pipelines add prompt-induced randomness on top, so consensus over prompts does not fully fix a brittle retriever. We are honest about the threat: decisive check one below directly compares our one-pass signal against K-pass variance, and if head dispersion fails to reach a usable correlation, the K-pass method wins on quality and the proposal's confidence component should be abandoned in favor of it.
Why not just fine-tune one large domain-specialized model end-to-end? Domain specialization with big models genuinely works — the stewardship study (Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation) found that serializing ICU records into pseudo-notes and embedding them with BioMegatron beat tabular LightGBM by a wide margin (ROC AUC 0.7610 versus 0.5431 for vancomycin susceptibility; 0.7644 versus 0.5083 for rifampin). But the same paper reports severe truncation from BioMegatron's 512-token window on long histories, and two results in the file argue that scale plus supervised fine-tuning is the wrong recipe here: AttriLens-Mol found supervised fine-tuning alone leaves small models at average scores around 45–49 while verifier-grounded RL reaches 64.7–67.9, beating 100B+ models; and LiT5 found its 3B variant overfit and underperformed its 770M variant, so capacity alone can backfire. Our design keeps the model compact, the context decoupled (Fusion-in-Decoder), and the training grounded — but if the end-to-end fine-tuned baseline matches our quality at acceptable latency in our own experiments, that result would stand and we would report it.
Why not just threshold the softmax probability or entropy to detect brittleness? One forward pass, one confidence number — the cheapest possible option, so it must be beaten. The evidence against it is specific: SENSEL shows MaxProb and entropy are poor correctness indicators because models can be overconfident and wrong, and Rethinking Hallucinations shows entropy and surprisal track output randomness rather than factual correctness, with ambiguity rates exceeding 50% across prompt formats on Med-HALT and roughly 60% for Llama2-13B-Chat. We include MaxProb and entropy as baselines in every experiment, and decisive check two requires our head-dispersion signal to beat them on the F1-coverage curve; if it does not, the cheap alternative is the right one and our confidence mechanism is not worth its complexity.
Proposed Experiments and Decisive Checks
Experiment one tests antimicrobial peptide retrieval. We construct queries from ESCAPE's five-class functional labels (antibacterial, antifungal, antiviral, antiparasitic, general antimicrobial) over its 21,409 AMPs and 60,950 non-AMPs, and evaluate on DRAMP 4.0 (DRAMP 4.0: an open-access data repository dedicated to the clinical translation of antimicrobial peptides, 30,260 entries) as an external test set, reporting mean average precision and Recall@100. DRAMP's hemolysis and cytotoxicity annotations let us test the attribute-scent mechanism on safety-constrained queries, which no re-ranking work in the file addresses. Experiment two tests zero-shot transfer to specialized literature retrieval on BEIR's BioASQ, SciFact, and COVID splits, with HyQE's reported Contriever-to-re-ranked improvement on COVID (27.32 to 35.54 NDCG@10) as the external reference point for what a re-ranking stage should add. Experiment three tests deployment: antibiotic susceptibility prediction on the MIMIC-IV cohort of 5,976 presumed-staph prescriptions from the stewardship study, where our system ranks candidate therapies with confidence, low-confidence cases abstain to a human, and we report the area under the F1-coverage curve against MaxProb, entropy, and the SENSEL-style K-pass upper bound.
Four checks decide the proposal. First, the correlation between one-pass head dispersion and 16-prompt sensitivity must be strong enough that the single-pass signal preserves most of the abstention value of the K-pass method; SENSEL's reported sensitivity-accuracy Pearson correlation of up to -0.40 gives a scale for what "strong enough" looks like. Second, the F1-coverage curve for our signal must exceed MaxProb and entropy on the stewardship task. Third, latency must remain one forward pass per candidate set — we report wall-clock and total FLOPs alongside every quality number, using REALM's reported 23.4–84.4% inference-call reduction (REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking) as the efficiency bar an uncertainty-aware re-ranker should clear. Fourth, the verifier-RL model must show a smaller accuracy drop under query paraphrase and candidate-order permutation than its supervised-only ablation — the direct test of whether grounded training reduces brittleness, the question that InteRank's ungrounded reward and AttriLens-Mol's RDKit-bound verifier both leave open in this setting.
References
- ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- AttriLens-Mol: Attribute Guided Reinforcement Learning for Molecular Property Prediction with Large Language Models
- Universal Biological Sequence Reranking for Improved De Novo Peptide Sequencing
- A Standardized Benchmark for Multilabel Antimicrobial Peptide Classification
- LiT and Lean: Distilling Listwise Rerankers Into Encoder-Decoder Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking
- HyQE: Ranking Contexts with Hypothetical Query Embeddings
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation
- DRAMP 4.0: an open-access data repository dedicated to the clinical translation of antimicrobial peptides
- REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking