Learning a New Clinician's Contouring Style from Pairwise Choices

Doctors draw different boundaries around the same tumor or organ, and many of these differences are consistent habits rather than random mistakes. One pathologist systematically calls borderline glands higher-grade. One radiation oncologist systematically draws wider margins. Recent segmentation models can represent these habits as a short "style vector" learned from a fixed pool of annotators. A clinician who was not in that pool, however, still receives either the consensus contour or a style that must be inferred from several expensive hand-drawn examples. We propose inferring a new clinician's style from something much cheaper than drawing: repeated choices of the better of two machine-generated contours. The system searches the model's style space by Bayesian optimization, a method that fits a probabilistic guess of the objective and uses it to pick the next most informative query.

The catch is dimensionality. A single 8-number style vector is probably too coarse, because a pathologist's bias can differ between each pair of adjacent grades, and an oncologist's margins can differ by structure. Once style is allowed to vary per class, the space reaches tens to low hundreds of dimensions. Preference-based Bayesian optimization is known to break down there. The core claim is twofold. First, a structured, higher-dimensional style space fits held-out clinicians measurably better than a single global vector. Second, recent numerical remedies for high-dimensional Bayesian optimization, together with a prior built from the population of already-known clinicians, make that space searchable with a clinically tolerable number of pairwise choices.

What exists and what is missing

Two lines of segmentation work motivate this proposal. The first treats disagreement as noise around a single truth. Disentangling Human Error from Ground Truth couples a segmentation network with per-annotator confusion matrices to recover one clean mask. The second treats disagreement as structured and meaningful. The Probabilistic U-Net and MoSE generate multiple plausible masks per image. Neither of these two families addresses personalization.

Two papers do, and they supply our generative machinery. Pionono gives each known rater a Gaussian in an 8-dimensional latent space. Its mean encodes the rater's systematic bias and its variance encodes the rater's inconsistency with themselves over time. The Latent Doctor Model encodes a new doctor's style from a support set of images that doctor has annotated. Pionono only represents raters seen in training. The Latent Doctor Model needs the new doctor to hand-annotate a support set. Neither uses sequential, actively chosen queries, and both fix a single low-dimensional style vector. Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle chooses among raters by downstream diagnostic value. It does not adapt to an individual user.

On the optimization side, Preferential Bayesian Optimization supplies the query model: it learns from pairwise "duels" rather than scores. That work also constrains our design. Its surrogate lives on pairs of inputs, which doubles the dimension, and it was demonstrated only on functions of one or two dimensions. Understanding High-Dimensional Bayesian Optimization (MSR) supplies the numerical fix we reuse. It shows that high-dimensional failures often come from length-scale initializations that freeze the likelihood gradient and from flat acquisition surfaces. It also shows that a global model with length scales initialized to scale with $\sqrt{d}$, combined with perturbations of a random subset of coordinates around the best point, rivals more elaborate methods.

TuRBO and ALEBO are the trust-region and linear-embedding alternatives we compare against. SI-BO motivates our use of a learned low-dimensional subspace. In our setting, that subspace comes from the spread of known clinicians rather than from the new clinician's early queries. To our knowledge, no work in this literature scales preferential optimization beyond a few dimensions or applies it to clinician style.

Core Claim

A clinician's contouring style can be recovered to near the level achievable from full annotation using a few dozen pairwise choices. Two conditions are required. The style space must be structured per class, not a single global vector. The preference search must avoid the product-space surrogate and adopt the numerical remedies that make high-dimensional Gaussian-process models trainable.

Research Program

Structured style model. We first build the generative model. We train a Pionono-style network whose style latent is split into one sub-vector per class, or per class-boundary, of 8 dimensions each. The total dimension $D$ is therefore 8 times the number of sub-vectors, in the range 32 to 128 depending on the dataset. A shared "gold" distribution serves as the consensus. The first question is representational and involves no preferences at all. For each held-out rater, we fit the latent directly to their training annotations by gradient descent and measure agreement on their unseen test images. This "oracle-fit" ceiling is computed for both the global 8-dimensional latent and the structured latent. If the structured latent gives no gain, the high-dimensional problem is not needed, and the proposal reduces to low-dimensional preference search. We treat that as a genuine outcome, not a failure to hide.

Preference search. We then build the search. We place a Gaussian process on an unknown utility $g(z)$ over style vectors $z \in \mathbb{R}^D$, not on pairs, so the surrogate stays in $D$ dimensions. We fit it with the logistic duel likelihood

$$ P(z \succ z') = \sigma\big(g(z) - g(z')\big), $$

where $\sigma$ is the logistic function. This is the same link PBO uses, applied to a utility function rather than a pair function.

Three additions address dimensionality. The first is MSR's $\sqrt{D}$-scaled length-scale initialization. The second is candidate generation by perturbing a random subset of coordinates around the current preferred style. The third, and the proposal's key ingredient, is a population prior. The Gaussian-process prior mean and the initial search subspace come from the known raters' latent means. We use their principal directions, with the known-rater Gaussians setting an initial trust region. Each query shows the clinician the two candidate styles rendered on a small panel of 3 to 5 images, and the clinician picks the one they would sign off on.

Simulated clinician. We simulate the clinician by leave-one-rater-out. The held-out rater prefers the candidate with higher agreement to their own annotations on the panel images. Choice noise is set by that rater's own intra-observer variance, estimated by the style model. This choice matters: a noiseless oracle would overstate how few queries are needed.

Baselines

The do-nothing reference is the consensus "gold" prediction of the same structured model, shown to the new clinician without any adaptation. Every method must beat it, and the gap between it and the oracle-fit ceiling is the quantity that personalization has to close. For context on this model family, Pionono reports an unweighted Cohen's kappa of 0.758 on Gleason 2019 and 0.711 on the 25-rater breast cancer set (Pionono). These numbers are not per-held-out-rater scores. We found no reported result on Lacuna for held-out-rater agreement, so we reproduce Pionono under our splits.

The strongest existing method is the conditional Latent Doctor Model, which infers style from a support set of $k$ hand-annotated images. It reports a KL divergence of 0.461 on Gleason 2019 in its own protocol. We run it on the same backbone, splits and held-out raters, with $k$ from 1 to 10. We report agreement as a function of both annotated images and pairwise choices, because the two kinds of effort are not interchangeable. Beating it means matching its agreement at $k = 3$ to $5$ with a number of pairwise choices whose plausible review time is smaller. We report the break-even ratio so readers can apply their own time costs.

The optimization baselines all share the same query budget, the same candidate-rendering panel and the same noise model. They are:

  • random search in the structured latent;
  • standard product-space PBO with dueling Thompson sampling (Preferential Bayesian Optimization);
  • the utility-space Gaussian process without our three additions;
  • a TuRBO-style trust-region variant (TuRBO);
  • CMA-ES, an evolution strategy that uses only the ranking of candidates, driven by the same pairwise choices.

The ablations isolate the population prior, the $\sqrt{D}$ initialization and the coordinate-subset perturbations one at a time. They also cross the global 8-dimensional latent against the structured latent under identical search.

Why not just…?

Why not just have the clinician draw a few contours and use the Latent Doctor Model? This is the most serious competitor. If the Latent Doctor Model at $k = 2$ matches our method at 40 choices, our method only wins when selecting a contour is much faster than drawing one. We cannot measure that ratio in simulation. The experiments therefore report the break-even ratio explicitly rather than assuming one. We also run a hybrid, one drawn image followed by choices, which may dominate both approaches.

Why not just pick the closest known rater? A multi-armed bandit over the training raters' styles is simple and could work whenever new clinicians resemble old ones. On datasets with four to six raters, it probably will not, because the held-out rater's oracle-fit style should lie outside the known styles. The 25-rater breast set is where this alternative is strongest. We include it as a baseline and report the fraction of held-out raters whose oracle-fit style is better than every known style.

Why not just search the 8-dimensional latent, where plain PBO already works? If the structured latent does not raise the oracle-fit ceiling, this alternative wins, and the first experiment is designed to reveal that before any preference search is run.

Why not just use CMA-ES on rankings? This alternative needs no surrogate. It beat Gaussian-process methods on the discontinuous 72-dimensional Hexapod task in ALEBO. We expect its advantage to vanish at budgets under a hundred choices, but the head-to-head comparison settles the question.

Why not just fine-tune the network on the clinician's corrections? This requires dense corrections, which again means drawing. It also risks overfitting a large network to a handful of images. It is included as a correction-budget-matched arm on Gleason 2019.

Proposed Experiments

The datasets are the breast cancer histopathology set with 25 raters as the primary benchmark, Gleason 2019 with 6 pathologists, and LIDC-IDRI with 4 radiologists as a small-pool stress test. All three appear in the file with multiple independent annotations per image. SegTHOR is excluded because it lacks multiple raters.

The metrics are:

  • Cohen's kappa for the histology sets and Dice for LIDC-IDRI, each measured against the held-out rater on their unseen images;
  • the fraction of the gap closed, $(A_t - A_0)/(A^* - A_0)$, where $A_t$ is agreement after $t$ choices, $A_0$ is the consensus agreement, and $A^*$ is the oracle-fit agreement;
  • the number of choices needed to close 80% of that gap.

Every configuration is repeated over all held-out raters with at least ten seeds.

Three checks decide the outcome. The first test asks whether the structured latent raises $A^*$ over the global 8-dimensional latent on the 25-rater set. If it does not, we stop the high-dimensional line. The second asks whether our search closes 80% of the gap in fewer choices than product-space PBO, CMA-ES and the closest-known-rater bandit, with the population-prior ablation showing where the gain comes from. The third asks whether the advantage persists when the simulated choice noise is doubled, which is the regime a real, tired clinician would present.