TL;DR
When evaluating medical image segmentation, the choice of which human annotation is used to tune a model's binary threshold can quietly skew reported performance. This paper demonstrates that trying to mathematically balance thresholds between disagreeing observers (a "maximin" strategy) fails to improve generalization on held-out data, underscoring the need for researchers to explicitly report both their threshold-selection and evaluation references.
What Problem Does This Paper Address?
Most automated retinal vessel segmentation models output a continuous probability map, but real-world evaluation requires a binary mask. To get this mask, researchers must select a decision threshold.
However, popular benchmarks like CHASE_DB1 provide annotations from multiple human experts (observers) who often disagree on fine details, such as where a tiny blood vessel ends. This creates a hidden experimental variable: which observer's annotations should be used to tune the threshold, and which should be used to calculate the final test score? If a threshold is tuned and tested against the same observer, it can artificially inflate performance by catering to that specific person's annotation style, masking how well the model actually generalizes.
Methods
To isolate the impact of threshold selection from model training, the authors established a highly controlled pipeline:
- Models: Standard Random Forests (RF) and Extra Trees (ET) trained on 29 handcrafted multiscale image features. Crucially, the models were trained solely against Observer 1's annotations.
- Cross-Validation: A strict, subject-separated 7-fold cross-validation protocol. Because both eyes of a subject share high anatomical similarity, the protocol ensures that both eyes of any given subject stay together in either the training, validation, or test split.
Using identical out-of-fold score maps generated by these models, the authors compared five global threshold policies:
- Fixed 0.50: An untuned baseline.
- O1-tuned: Optimized to maximize the Dice score against Observer 1.
- O2-tuned: Optimized against Observer 2.
- Mean-tuned: Optimized against the average Dice score of both observers.
- Maximin: Optimized to maximize the lower (worst-case) per-image Dice score between the two observers.
Datasets
The study uses the complete, native-resolution CHASE_DB1 dataset, which consists of 28 digital fundus photographs from 14 pediatric subjects, complete with two independent sets of human expert annotations.
Key Insights and Ideas
- The "Maximin" Overfitting Trap: Intuitively, optimizing a threshold to satisfy the more conservative or critical observer (maximin) sounds like a robust strategy. However, the authors found that this objective overfits to the specific annotation discrepancies present in the small validation set (only 4 images per fold), failing to translate into better performance on unseen test subjects.
- Reference Dependence is Massive: The exact same binary mask yields drastically different performance metrics depending purely on who it is evaluated against. This means a "high-performing" model might just be a model whose threshold was lucky enough to match the test observer's bias.
Key Results
- Maximin vs. Single-Observer Tuning: For Random Forests, switching from an Observer-1-tuned threshold to a maximin threshold changed the decision boundary in 19 out of 21 fits. Despite this frequent adjustment, the primary metric $W$ (the mean of the lower observer Dice per image) slightly decreased from 70.53% to 70.45%.
- Statistical Insignificance: The paired difference between maximin and O1-tuning was a negligible -0.073 percentage points, with a 95% bootstrap confidence interval spanning zero ([-0.384, 0.238]), showing no empirical benefit for the more complex policy.
- The Evaluation Discrepancy: Identical masks generated by an O1-tuned Random Forest scored 73.66% against Observer 1, but dropped to 71.06% when evaluated against Observer 2. This ~2.6% gap is purely a function of evaluation reference choice, not model architecture.
Limitations and Future Work
The authors acknowledge several constraints in this study:
- Cohort Size: The evaluation is limited to the 14 subjects within the CHASE_DB1 dataset.
- Model Architecture: The study relies on traditional tree ensembles with handcrafted features rather than modern deep learning architectures (like U-Nets or Transformers).
- Asymmetric Training: Models were only trained using Observer 1 as the ground truth. The dynamics might shift if models are trained on fused labels or subjected to a symmetric crossover training design against Observer 2.
Summary updated