TL;DR
When medical image datasets are split randomly by "crop" rather than by the original microscope slide or field, models can "cheat" by recognizing field-specific artifacts like stain and illumination. This paper demonstrates that such "acquisition leakage" significantly inflates performance on cervical cytology benchmarks—by as much as 12 percentage points—and proves that common diagnostic tools like Grad-CAM and linear probes are unable to detect or fix this shortcut learning.
What Problem Does This Paper Address?
In medical imaging, data is often hierarchical: one patient has multiple slides, one slide has multiple microscope fields, and one field contains many individual cell crops. Many public benchmarks are released as collections of these crops. The "default" behavior for many researchers is to perform a random 80/20 split at the crop level.
However, this creates a hidden problem: acquisition leakage. Sibling crops from the same microscope field share identical "acquisition parents"—meaning they share the same stain intensity, focus, illumination, and background textures. If one sibling is in the training set and another is in the test set, the model can learn to identify the "field" rather than the biological features of the cell. The authors set out to measure exactly how much this inflates scores and whether standard "explainability" tools can warn us when it's happening.
Methods
The researchers conducted a rigorous, preregistered study involving 2,281 trained models across twelve different architectures (from ResNet to Vision Transformers). Their approach featured:
- Protocol Comparison: They compared a standard crop-level split (P1) against a "parent-disjoint" split (P2), where all crops from a single field or slide stay together on one side of the split.
- Causal Isolation (Factorial Design): To ensure a lower score in P2 wasn't just due to having fewer unique "parents" in the training set (a diversity confound), they designed a factorial experiment that fixed the number of training crops and parents, varying only whether the test set's "siblings" were present in training.
- Diagnostic Interventions: They tested whether the leak could be "erased" using linear concept erasure (LEACE), gradient reversal, or color jittering, and whether Grad-CAM saliency maps could distinguish between "leaking" and "clean" models.
Datasets
The study audited four public cervical cytology datasets:
- SIPaKMeD: 4,049 crops from 966 fields.
- CRIC: 11,534 cells from 400 slide images.
- Cx22: 1,320 regions from 514 source images (structural audit only).
- Herlev: 917 images with one cell per image (used as a negative control, as it has no siblings).
Key Insights and Ideas
- The "Diversity" Myth: A common defense for crop-level splitting is that it provides more "diversity" in the training set. The authors' factorial experiment debunked this for SIPaKMeD: the performance gain came entirely from seeing siblings of the test cells, not from the increased variety of parent fields.
- Diagnostics are Blind: This is perhaps the most alarming finding. Even when the researchers erased 82% of the "decodable" parent identity from the model's features, the performance gap remained. Similarly, Grad-CAM maps—often used to "verify" that a model is looking at the right thing—looked identical for both leaking and clean models.
- Color is the Culprit: The only intervention that significantly reduced the leak was matching a test cell's color statistics to an unseen field of the same class. This suggests the models are latching onto field-specific appearance (stain/lighting) as a shortcut.
- Rankings Persist, Accuracies Don't: While the relative ranking of different model architectures stayed mostly the same, the absolute accuracy numbers were non-comparable. On the CRIC dataset, the "protocol gap" (12pp) was larger than the difference between the best and worst architectures.
Key Results
- SIPaKMeD: Crop-level splitting inflated accuracy by 0.97 percentage points (95% CI 0.59–1.39). While small, the factorial experiment (Fig 2c) showed this was purely due to leakage (+1.73 points).
- CRIC: The inflation was massive—12.08 percentage points (11.16–12.99). Balanced accuracy plummeted from ~78% to ~59% when moving to a proper parent-disjoint split.
- Decodability: Parent identity was decodable at 100x chance, but "fixing" this decodability did not fix the leak, proving that "probes establish availability, not use."
Limitations and Future Work
- Deepest Released Units: The authors could only split by the "parents" identified in the datasets (fields or slides). They note that "patient-level" grouping might reveal even deeper dependencies and further leakage that they couldn't measure here.
- Mechanism Identification: While they narrowed the mechanism down to field-specific appearance (via color transfer), they haven't fully identified exactly what nonlinear or spatial features the networks are learning to recognize specific fields.
- Recommendations: The authors urge dataset releasers to label every observation with its parent/patient ID and recommend that evaluators always split at the deepest released unit and report the "sibling fraction" in their test sets.
Summary updated