TL;DR
Glaucoma detection models that achieve near-perfect accuracy on their training data often fail significantly when deployed to new hospitals or cameras. This study quantifies a massive "performance cliff"—a drop in mean AUC from 0.91 to 0.70—and demonstrates that while color normalization and strong data augmentation can mitigate this gap, they are not a complete cure for dataset bias.
What Problem Does This Paper Address?
In medical AI research, it is common to report stellar performance (AUC > 0.95) for models trained and tested on the same dataset. However, this "in-domain" success rarely translates to the real world. When a model trained at one clinic is used at another, it encounters different cameras, lighting conditions, patient populations, and even different expert labeling habits.
This paper investigates the cross-dataset generalization gap in glaucoma detection. The authors aim to determine if specific CNN architectures are more robust than others and whether common techniques like multi-source training or image normalization can bridge the performance gap.
Methods
The researchers conducted a rigorous, controlled experiment:
- Architectures: Five standard ImageNet-pretrained CNNs (ResNet50, DenseNet121, MobileNetV2, EfficientNet-B0, and InceptionV3) were fine-tuned using identical settings.
- Training Protocol: Models were trained on one dataset and tested both "in-domain" (on a held-out portion of the same dataset) and "cross-source" (on entirely different datasets).
- Interventions: They tested several strategies to improve generalization:
- Threshold Recalibration: Resetting the decision threshold using a small (10%) sample of the new target data.
- Multi-Source Training: Training on a pool of multiple datasets.
- Appearance Normalization: Comparing five methods, including Gray-world balancing, Ben Graham's local subtraction, CLAHE, and LAB reference normalization.
- Strong Color Augmentation: Using wider ranges of brightness, contrast, saturation, and hue during training.
Datasets
The study utilized four public fundus image datasets, which vary significantly in appearance and cropping (see Figure 1):
- ACRIMA: 705 images ( FISABIO, Spain).
- ORIGA: 650 images (Singapore).
- RIM-ONE DL: 485 images (three Spanish hospitals).
- G1020: 1020 images (used as a "stress test" external target).

Key Insights and Ideas
- The "Performance Cliff" is Real: Every single model architecture suffered a major drop in performance when moving to a new dataset. No specific architecture emerged as a "silver bullet" for robustness.
- Thresholds Don't Travel: A decision threshold optimized for one dataset is often disastrous on another. For example, models trained on ACRIMA and tested on RIM-ONE DL tended to label almost every image as "glaucoma," causing specificity to plummet from 0.87 to 0.51.
- Color is a Major Culprit: The most effective remedies were those that addressed color and illumination differences. LAB reference normalization and strong color augmentation reduced the performance drop by about 26%.
- More Data Isn't Always Better: Surprisingly, simply pooling two datasets (multi-source training) did not reliably improve performance on a third, unseen dataset compared to single-source training.
- The G1020 Mystery: All models performed at chance level (AUC ~0.5) on the G1020 dataset, regardless of training. Even models trained within G1020 failed, suggesting issues with label quality or that the features used for glaucoma in other datasets were simply absent or unextractable here.
Key Results
- AUC Drop: The mean in-domain AUC was 0.913, which crashed to 0.702 in cross-source testing—a drop of 0.211.
- Best Performers: LAB reference normalization (cross-source AUC 0.755) and strong color augmentation (0.754) provided the best gains. Figure 2 illustrates how these normalization methods attempt to standardize the "look" of the images.
- Recalibration: Resetting the threshold on just 10% of target images improved specificity (0.51 to 0.67) but at the cost of sensitivity (0.73 to 0.61).

The stark reality of the generalization gap is best summarized by the comparison of in-domain vs. cross-source AUCs across all models (Figure 5).

Limitations and Future Work
The authors acknowledge several limitations: the datasets are relatively small, and "patient leakage" (both eyes of one patient appearing in both training and test sets) might have inflated in-domain results. Furthermore, they used fixed hyper-parameters rather than per-model tuning.
Future research should explore more advanced domain adaptation techniques that go beyond simple color normalization. The authors conclude with a vital practical recommendation: any medical AI model must be validated on external data from the intended deployment site, and its operating threshold must be recalibrated locally before clinical use.
Summary updated