Observer Choice and Threshold Selection in Retinal Vessel Segmentation: A Subject-Separated Evaluation

Wenhao Xu, Yixian Kong, Ting Pan, Changwei Wang, Feilong Wang, Rongtao XuarXiv 2026 · arXiv

TL;DR

When evaluating medical image segmentation, the choice of which human annotation is used to tune a model's binary threshold can quietly skew reported performance. This paper demonstrates that trying to mathematically balance thresholds between disagreeing observers (a "maximin" strategy) fails to improve generalization on held-out data, underscoring the need for researchers to explicitly report both their threshold-selection and evaluation references.

What Problem Does This Paper Address?

Most automated retinal vessel segmentation models output a continuous probability map, but real-world evaluation requires a binary mask. To get this mask, researchers must select a decision threshold.

However, popular benchmarks like CHASE_DB1 provide annotations from multiple human experts (observers) who often disagree on fine details, such as where a tiny blood vessel ends. This creates a hidden experimental variable: which observer's annotations should be used to tune the threshold, and which should be used to calculate the final test score? If a threshold is tuned and tested against the same observer, it can artificially inflate performance by catering to that specific person's annotation style, masking how well the model actually generalizes.

Methods

To isolate the impact of threshold selection from model training, the authors established a highly controlled pipeline:

  • Models: Standard Random Forests (RF) and Extra Trees (ET) trained on 29 handcrafted multiscale image features. Crucially, the models were trained solely against Observer 1's annotations.
  • Cross-Validation: A strict, subject-separated 7-fold cross-validation protocol. Because both eyes of a subject share high anatomical similarity, the protocol ensures that both eyes of any given subject stay together in either the training, validation, or test split.
Figure 1: The fixed seven-fold allocation. Each column represents a subject and both eyes. The four validation images select the threshold; the four test images are scored only after that choice is recorded.

Using identical out-of-fold score maps generated by these models, the authors compared five global threshold policies:

  1. Fixed 0.50: An untuned baseline.
  2. O1-tuned: Optimized to maximize the Dice score against Observer 1.
  3. O2-tuned: Optimized against Observer 2.
  4. Mean-tuned: Optimized against the average Dice score of both observers.
  5. Maximin: Optimized to maximize the lower (worst-case) per-image Dice score between the two observers.

Datasets

The study uses the complete, native-resolution CHASE_DB1 dataset, which consists of 28 digital fundus photographs from 14 pediatric subjects, complete with two independent sets of human expert annotations.

Key Insights and Ideas

  • The "Maximin" Overfitting Trap: Intuitively, optimizing a threshold to satisfy the more conservative or critical observer (maximin) sounds like a robust strategy. However, the authors found that this objective overfits to the specific annotation discrepancies present in the small validation set (only 4 images per fold), failing to translate into better performance on unseen test subjects.
  • Reference Dependence is Massive: The exact same binary mask yields drastically different performance metrics depending purely on who it is evaluated against. This means a "high-performing" model might just be a model whose threshold was lucky enough to match the test observer's bias.

Key Results

  • Maximin vs. Single-Observer Tuning: For Random Forests, switching from an Observer-1-tuned threshold to a maximin threshold changed the decision boundary in 19 out of 21 fits. Despite this frequent adjustment, the primary metric $W$ (the mean of the lower observer Dice per image) slightly decreased from 70.53% to 70.45%.
  • Statistical Insignificance: The paired difference between maximin and O1-tuning was a negligible -0.073 percentage points, with a 95% bootstrap confidence interval spanning zero ([-0.384, 0.238]), showing no empirical benefit for the more complex policy.
Figure 2: Every subject’s change in worst-observer Dice, after averaging both eyes and three seeds. Positive values favor maximin; negative values favor O1-tuned. Subject IDs follow the public data filenames.
  • The Evaluation Discrepancy: Identical masks generated by an O1-tuned Random Forest scored 73.66% against Observer 1, but dropped to 71.06% when evaluated against Observer 2. This ~2.6% gap is purely a function of evaluation reference choice, not model architecture.
Qualitative examples showing subtle differences in final masks between O1-tuned and Maximin policies near the optic disc.

Limitations and Future Work

The authors acknowledge several constraints in this study:

  • Cohort Size: The evaluation is limited to the 14 subjects within the CHASE_DB1 dataset.
  • Model Architecture: The study relies on traditional tree ensembles with handcrafted features rather than modern deep learning architectures (like U-Nets or Transformers).
  • Asymmetric Training: Models were only trained using Observer 1 as the ground truth. The dynamics might shift if models are trained on fused labels or subjected to a symmetric crossover training design against Observer 2.

Summary updated

Abstract

The annotation used to select a segmentation threshold is part of the evaluation protocol, yet its effect is easily conflated with model quality. We examine this choice for retinal vessel segmentation using all 28 CHASE DB1 images and both human annotations. A fixed seven-fold protocol keeps both eyes of each of the 14 subjects together. Random forests and Extra Trees are fitted against observer 1 with three random seeds, yielding 42 fits. Five threshold policies share identical score maps: fixed 0.50, observer-1 tuning, observer-2 tuning, mean-observer tuning, and maximin tuning of the per-image lower observer Dice. For random forests, maximin changes the threshold in 19 of 21 fits, but worst-observer Dice decreases from 70.53 percent to 70.45 percent. The paired difference is -0.073 percentage points, with a conditional subject-bootstrap 95 percent interval of [-0.384, 0.238]. Extra Trees shows the same direction. Identical observer-1-tuned random-forest masks score 73.66 percent against observer 1 and 71.06 percent against observer 2. The results support explicit reporting of both the threshold-selection reference and evaluation reference; they do not support an accuracy benefit from maximin tuning in this cohort. All splits, raw predictions, metrics and code are supplied. AI assistance is disclosed.
BibTeX
@misc{xu2026observer,
  author = {Wenhao Xu and Yixian Kong and Ting Pan and Changwei Wang and Feilong Wang and Rongtao Xu},
  title = {{Observer Choice and Threshold Selection in Retinal Vessel Segmentation: A Subject-Separated Evaluation}},
  year = {2026},
  eprint = {2609.25597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.25597},
}
Figure