TL;DR
ReuScope is a new auditing protocol that disentangles data reuse (leakage) from model performance sensitivity. By applying it to the Capsule Vision 2024 challenge, the authors show that "leakage" doesn't always mean "inflated scores"—in fact, removing reused data can sometimes improve performance on certain subsets of the data, revealing that source composition often matters more than individual duplicate instances.
What Problem Does This Paper Address?
In machine learning, "data leakage"—where training data accidentally ends up in the validation or test set—is a persistent headache. In medical imaging, this is particularly tricky: a single patient might have multiple videos, and a single video has thousands of highly similar frames. Simply checking for identical filenames isn't enough, as the same image might appear under different names or slightly different labels across datasets.
Existing audits often stop at detecting reuse (e.g., finding duplicates). However, the authors argue that detecting reuse is a different task from measuring its effect on a model's score. If you "fix" a dataset by removing a source that contains leaked data, you aren't just removing duplicates; you're also removing legitimate training signals, changing the class balance, and reducing the total sample size. This makes it nearly impossible to tell if a score drop is due to "fixing leakage" or simply making the training set worse.
Methods
The authors introduce ReuScope, a four-step protocol for auditing benchmarks:
- Define Boundaries: Explicitly state which populations are being compared (e.g., public training vs. validation).
- Measure Relationships: Use a hierarchy of checks, including cryptographic hashes (SHA-256) for byte-identical files, perceptual hashes (pHash, dHash) for near-matches, and video-prefix matching to find frames from the same recording.
- Separate Training and Evaluation Changes: This is the core of ReuScope. It uses two distinct comparisons:
- $\Delta_{view}$: Hold the model fixed and change the evaluation population (e.g., evaluate on a "clean" subset).
- $\Delta_{train}$: Hold the evaluation set fixed and change the training data (e.g., train with vs. without a specific source).
- Report with Scope and Uncertainty: Connect every observation to a specific supported conclusion, acknowledging what cannot be known (like unique patient counts when IDs are missing).
Datasets
The primary case study is Capsule Vision 2024 (CV2024), a challenge for abnormality classification in video capsule endoscopy. It aggregates data from four sources:
- KVASIR (Kvasir-Capsule): The largest source, contributing ~72% of validation rows.
- SEE-AI, KID, and AIIMS: Smaller, diverse sources.
The authors also apply parts of the protocol to other major datasets like ChestX-ray14, ISIC 2019, and HyperKvasir to demonstrate its generalizability.
Key Insights and Ideas
- The "Sign Reversal" Phenomenon: The most striking finding was that removing the KVASIR source from training caused a massive drop in Balanced Accuracy (BA) on the full public validation set. However, when evaluating those same models only on the non-KVASIR validation rows, the models trained without KVASIR actually performed better. This suggests the "leakage" was providing domain-specific benefits that didn't generalize to other sources.
- Identity $\neq$ Relationship: Filename matching is a poor auditor. The authors found 216 groups of byte-identical files crossing the train-validation boundary, many of which had different names. More importantly, even without byte-identity, all 41 Kvasir-Capsule video recordings were present in both splits, meaning the model was effectively seeing "different frames of the same movie" during both training and testing.
- Composition Over Memorization: Through matched controls (adjusting for sample size and class counts), the authors showed that the performance changes were largely driven by source composition and class support, rather than the model simply "memorizing" specific leaked images.
Key Results
- Public Validation Drop: Removing all KVASIR training rows (and a few near-matches) lowered public-validation Balanced Accuracy from 82.5% to 61.2%.
- The Reversal: Evaluating these same models on non-KVASIR validation rows showed an increase of 1.38 percentage points in BA after removing the KVASIR training data.
- Internal Filtering: When they created an "Internal-filtered" validation set by removing any row that had a same-source training match (by filename or pHash), the performance difference between the "Full" and "Removal" models vanished (the 95% confidence interval included zero).
Limitations and Future Work
The authors acknowledge several limitations:
- Pipeline Specificity: The audit primarily focused on a DINOv2-L/14 backbone; results might vary with different architectures.
- Missing Metadata: The lack of cross-recording patient identifiers in the public CV2024 data means they could not definitively quantify patient-level overlap, which is the "gold standard" for medical leakage.
- Screening vs. Detection: Their similarity screens are intended as auditing tools, not perfectly calibrated duplicate detectors.
Future work could involve applying ReuScope to a wider variety of model families and advocating for better metadata standards in public medical imaging benchmarks to allow for true patient-level auditing.
Summary updated