Telling Subtle Diagnostic Pairs from Noise in Expert-Domain Preference Training

When a vision-language model is trained with preference pairs, meaning a better and a worse answer to the same image and question, it learns most from pairs whose difference is real and visible. In pathology or radiology, though, the most valuable pairs are often the ones whose difference is tiny: "usual interstitial pneumonia" versus "non-specific interstitial pneumonia" can hinge on a single morphological cue. The automated judges used to score such pairs are web-trained image-text models or general reward models. Seen through those judges, these subtle but correct pairs look identical to pairs that are just noisy, mislabeled, or invisible to the judge. Current practice cannot tell the two apart. One camp discards all low-margin pairs as noise. The other keeps them and loosens regularization.

We propose to score each pair not with one global similarity number but with a breakdown over the concrete visual findings that the two answers disagree about. Following ViP, these are plain-language descriptions such as "honeycombing" or "patchy uniform fibrosis" that even general models can partly see. A pair counts as clear when the verifier confidently separates the two answers on at least one finding where they differ, even if the total gap is small. It counts as noisy when the gap is spread thin across findings the verifier cannot resolve. Clear pairs are kept, and each pair's regularization strength is set from its total margin. Noisy pairs are dropped. The core claim is testable: in expert domains, low-margin pairs split cleanly into helpful and harmful groups once the margin is broken down by finding, and this split explains the conflicting results in the general preference-learning literature.

Where this sits

Three lines of work motivate the proposal. Less is More shows that near-zero-margin pairs shrink DPO updates. DPO, or direct preference optimization, is the standard method that trains a model directly on chosen-versus-rejected pairs without a separate reward model. That paper's BeeS selector keeps the top 10% of pairs by margin and gains 3–8% AlpacaEval 2.0 win rate over training on all data. β-DPO argues the opposite: low-gap pairs are the most informative if the regularization strength β is lowered. It reports 57.07% versus 51.51% win rate for vanilla DPO on Pythia-2.8B. Neither paper gives a criterion for which low-margin pairs are which, and both assume a judge that can see the relevant difference. Our proposal supplies that criterion. It also replaces β-DPO's batch-level β with a per-pair setting.

In multimodal hallucination, CLIP-DPO ranks candidate captions by CLIP image-text similarity and trains on high-contrast pairs, reaching an AMBER score of 89.6 on LLaVA-1.5. Its judge is a single global score from web-trained CLIP. TPO finds image-dependent tokens by adding Gaussian noise to the image. Diagnostic texture does not survive that kind of corruption, so its anchoring signal is unlikely to transfer to pathology. In the expert-domain setting, EAGLE builds pathology preference pairs by masking diagnostic regions and swapping medical entities, then trains with iterative DPO. It reports 66.42% Quilt-VQA recall against 56.60% for STLLaVA-Med. EAGLE never measures how clear its pairs are, and its authors report plateaus and overfitting in later iterations. We take EAGLE's pair generator as our pair pool and test whether that plateau is caused by unclear pairs.

The method supplier is ViP. It has an LLM rewrite disease names as visual symptoms, checks those symptoms against images with visual question answering, and improves zero-shot CLIP skin-lesion accuracy by 18.73%. ViP uses this only for closed-set classification prompts. We reuse the decomposition as the inside of a preference judge for free-form generation. Two papers constrain the design. MedVH shows that domain-tuned models hallucinate more readily on wrong images and false premises, a failure it calls the "Expertise Trap." Any domain alignment method must therefore be checked for worsening it. MERLIM shows answers swing by up to 26.5% under prompt paraphrase. We turn that fragility into a measurement: the verifier's spread across paraphrases serves as its uncertainty.

Core Claim

For a pair with chosen answer $y_w$ and rejected answer $y_l$ on image $x$, we extract the diagnostic claims each answer makes using medical entity recognition, as EAGLE does. We map each claim to its symptom list and keep the set $K$ of symptoms that the two answers disagree about. For each symptom $k$ in $K$, a verifier gives a score $s_k$ for how strongly the image supports that symptom under each answer. The verifier is either general CLIP or the domain model QuiltNet, applied to the symptom phrase. Averaging over $P$ prompt paraphrases gives a mean difference $\Delta_k$ and a paraphrase standard deviation $\sigma_k$. We define

$$ M = \sum_{k \in K} \Delta_k, \qquad C = \max_{k \in K} \frac{\Delta_k}{\sigma_k + \epsilon}, $$

where $M$ is the total margin, $C$ is the clarity (the most confidently resolved finding in favor of the chosen answer), and $\epsilon$ is a small constant. Pairs with $C$ below a threshold $\tau$ are dropped. Retained pairs are trained with a per-pair $\beta = \beta_0 \cdot g(M)$, where $g$ is an increasing function, so subtle-but-clear pairs get small β and large-gap pairs get large β, following β-DPO's reasoning. The claim is that ranking pairs by $C$ rather than $M$ is what matters, and that this matters more as the domain gap grows.

Research Program

The work has three parts. First, we build the symptom judge for histopathology. This means generating symptom lists for the diagnostic entities that appear in Quilt-VQA and PathVQA questions using an LLM, filtering them with ViP's question-answering check, and computing $\Delta_k$ and $\sigma_k$ with five paraphrases per symptom.

Second, we run a controlled pair-pool study. Starting from a Quilt-LLaVA-style 7B model after supervised fine-tuning, we generate one fixed pool of EAGLE-style pairs (masked-region and entity-swap negatives). Every method then trains on this same pool for the same number of DPO steps, over up to four self-training iterations. The direct test of the BeeS-versus-β-DPO conflict works as follows. We bin pairs by total margin $M$, and within the lowest-margin bin we split them by clarity $C$. We then train with each half added or removed. If low-margin high-clarity pairs help while low-margin low-clarity pairs hurt, both camps were each seeing half of a mixture.

Third, we test transfer. The same pipeline is applied to chest radiographs (IU-Xray, MIMIC-CXR), where specialist judges are weaker. A natural-image control on RLHF-V and AMBER checks whether the decomposition is useless or harmful when the domain gap is small, which is what we predict.

Baselines

The naive reference point is the supervised fine-tuned model with no preference training. It is also run with MedVH's skepticism hint added to the prompt, so the cheapest behavioral fix is represented. The direct prior system is EAGLE: vanilla iterative DPO on the full unfiltered pair pool. Its reported 66.42% Quilt-VQA recall is the number to beat, though our re-run on the shared pool is what we will actually compare against.

The strongest competitors on the core question are judge-based filters. CLIP-DPO-style filtering ranks pairs by global CLIP similarity and is run twice: once with general CLIP (reported AMBER 89.6 in the natural domain; no reported pathology result found on Lacuna) and once with QuiltNet from Quilt-1M, which reaches 93.28% zero-shot accuracy on LC25000. The QuiltNet judge is the most dangerous baseline because it attacks the domain gap directly. BeeS-style selection combines the policy's implicit margin with a QuiltNet score in BeeS's Bayesian way and keeps the top fraction. β-DPO runs on the full pool with its batch-adaptive β and 3σ outlier filter.

All filters are matched on the number of retained pairs. We also report an unmatched-count variant for each. Each method gets the same grid of 8 settings over its key hyperparameter (threshold, β, or top fraction), selected on a held-out validation split. Three ablations isolate our ingredients. The first uses the decomposed judge but filters on total $M$ alone, separating decomposition from clarity. The second uses clarity filtering with a fixed β. The third replaces each pair's symptom set with random symptoms drawn from other entities, testing whether the content of the decomposition matters. Beating the field means a statistically significant gain, measured by bootstrap over test questions, over the best of the QuiltNet-CLIP-DPO and BeeS baselines on both Quilt-VQA recall and a subtle-differential slice, with no loss on MedVH.

Why not just…?

Why not just use QuiltNet as the judge in CLIP-DPO? This is a real threat. A domain image-text model may close most of the semantic gap by itself. It still produces one global score, so a small QuiltNet gap cannot distinguish "differs only in honeycombing, clearly visible" from "the judge cannot tell." The QuiltNet-CLIP-DPO baseline, together with the decomposed-but-margin-only ablation, settles this question. If QuiltNet global filtering matches our method on the subtle slice, the claim fails and the contribution reduces to a better judge rather than a clarity criterion.

Why not just apply BeeS-style top-margin selection to EAGLE's pairs? BeeS assumes low-margin pairs are noise. That assumption would throw away exactly the subtle differential pairs that pathology alignment needs. The within-bin split experiment directly measures whether low-margin, high-clarity pairs carry useful signal that BeeS would discard.

Why not just run β-DPO and keep everything? β-DPO lowers β for an entire batch based on the average gap. It therefore loosens regularization on noisy pairs as readily as on informative ones, and its outlier filter targets only extreme pairs. The β-DPO baseline and the clarity-with-fixed-β ablation together show whether the per-pair filter adds anything beyond adaptive β.

Why not just steer activations at inference, as MEDA does? MEDA reports +10.6% on IU-Xray and +6.9% on MIMIC-CXR using Radiopaedia-derived steering vectors. Steering requires access to the model's hidden states and leaves the weights unchanged. In the radiology transfer we report MEDA-style steering as a complementary reference and test whether it stacks with our trained model, rather than claiming to dominate it.

Proposed Experiments and Decisive Checks

The main evaluations are Quilt-VQA recall and PathVQA (yes/no accuracy and open-ended F1, with older baselines of 68.2% and 24.0% from PathVQA). We add a held-out subtle-differential slice in which the candidate diagnoses share most symptoms, plus long-form CHAIR-style hallucination rates measured against entity lists. On MedVH, which tests wrongful images, false premises, and none-of-the-above questions, no scalar result beyond a report-generation CHAIR rate near 50% is found on Lacuna. We therefore report the full Characterization Score for every method. The Expertise Trap check is decisive: a method that gains recall but drops on MedVH's wrongful-image task is counted as failing. For radiology we use IU-Xray and MIMIC-CXR, and AMBER and POPE serve as the natural-domain control.

Three results would confirm the proposal. First, within the lowest-margin bin, adding high-clarity pairs should help and adding low-clarity pairs should hurt. Second, the advantage of $C$ over $M$ should be large in pathology, smaller in radiology, and near zero on AMBER, tracking the domain gap. Third, clarity filtering should remove EAGLE's late-iteration plateau, meaning recall keeps rising or holds through iteration four where unfiltered DPO declines. If the random-symptom ablation performs as well as the real decomposition, or QuiltNet global filtering matches us on the subtle slice, we will report that the clarity criterion does not survive.