ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024

Eunseob Choi, Kyeonghun Kim, Hyuk-Jae Lee, Nam-Joon KimarXiv 2026 · arXiv

TL;DR

ReuScope is a new auditing protocol that disentangles data reuse (leakage) from model performance sensitivity. By applying it to the Capsule Vision 2024 challenge, the authors show that "leakage" doesn't always mean "inflated scores"—in fact, removing reused data can sometimes improve performance on certain subsets of the data, revealing that source composition often matters more than individual duplicate instances.

What Problem Does This Paper Address?

In machine learning, "data leakage"—where training data accidentally ends up in the validation or test set—is a persistent headache. In medical imaging, this is particularly tricky: a single patient might have multiple videos, and a single video has thousands of highly similar frames. Simply checking for identical filenames isn't enough, as the same image might appear under different names or slightly different labels across datasets.

Existing audits often stop at detecting reuse (e.g., finding duplicates). However, the authors argue that detecting reuse is a different task from measuring its effect on a model's score. If you "fix" a dataset by removing a source that contains leaked data, you aren't just removing duplicates; you're also removing legitimate training signals, changing the class balance, and reducing the total sample size. This makes it nearly impossible to tell if a score drop is due to "fixing leakage" or simply making the training set worse.

Methods

The authors introduce ReuScope, a four-step protocol for auditing benchmarks:

  1. Define Boundaries: Explicitly state which populations are being compared (e.g., public training vs. validation).
  2. Measure Relationships: Use a hierarchy of checks, including cryptographic hashes (SHA-256) for byte-identical files, perceptual hashes (pHash, dHash) for near-matches, and video-prefix matching to find frames from the same recording.
  3. Separate Training and Evaluation Changes: This is the core of ReuScope. It uses two distinct comparisons:
  4. $\Delta_{view}$: Hold the model fixed and change the evaluation population (e.g., evaluate on a "clean" subset).
  5. $\Delta_{train}$: Hold the evaluation set fixed and change the training data (e.g., train with vs. without a specific source).
  6. Report with Scope and Uncertainty: Connect every observation to a specific supported conclusion, acknowledging what cannot be known (like unique patient counts when IDs are missing).
Figure 1: Overview of data relationships and training comparisons across evaluation views in ReuScope. The protocol audits file identity, shared videos and similarity candidates across data splits. Paired Full and Removal models are evaluated on three validation populations, separating training changes from evaluation changes. The resulting performance contrasts demonstrate evaluation- population sensitivity rather than establishing score inflation from data reuse.

Datasets

The primary case study is Capsule Vision 2024 (CV2024), a challenge for abnormality classification in video capsule endoscopy. It aggregates data from four sources:

  • KVASIR (Kvasir-Capsule): The largest source, contributing ~72% of validation rows.
  • SEE-AI, KID, and AIIMS: Smaller, diverse sources.

The authors also apply parts of the protocol to other major datasets like ChestX-ray14, ISIC 2019, and HyperKvasir to demonstrate its generalizability.

Key Insights and Ideas

  • The "Sign Reversal" Phenomenon: The most striking finding was that removing the KVASIR source from training caused a massive drop in Balanced Accuracy (BA) on the full public validation set. However, when evaluating those same models only on the non-KVASIR validation rows, the models trained without KVASIR actually performed better. This suggests the "leakage" was providing domain-specific benefits that didn't generalize to other sources.
  • Identity $\neq$ Relationship: Filename matching is a poor auditor. The authors found 216 groups of byte-identical files crossing the train-validation boundary, many of which had different names. More importantly, even without byte-identity, all 41 Kvasir-Capsule video recordings were present in both splits, meaning the model was effectively seeing "different frames of the same movie" during both training and testing.
  • Composition Over Memorization: Through matched controls (adjusting for sample size and class counts), the authors showed that the performance changes were largely driven by source composition and class support, rather than the model simply "memorizing" specific leaked images.

Key Results

  • Public Validation Drop: Removing all KVASIR training rows (and a few near-matches) lowered public-validation Balanced Accuracy from 82.5% to 61.2%.
  • The Reversal: Evaluating these same models on non-KVASIR validation rows showed an increase of 1.38 percentage points in BA after removing the KVASIR training data.
  • Internal Filtering: When they created an "Internal-filtered" validation set by removing any row that had a same-source training match (by filename or pHash), the performance difference between the "Full" and "Removal" models vanished (the 95% confidence interval included zero).
Figure 3: A training comparison reverses across evaluation populations (DINOv2-L/14). Full uses 37,607 rows; Removal excludes KVASIR and five other-source rows passing the Kvasir-Capsule hash screen, retaining 10,591. Markers show ten-seed mean BA; faint paths follow each fitted model’s scores across views. Full leads on public validation; Removal has higher mean BA on non-KVASIR. Below each view are paired Removal-minus-Full changes in percentage points and pointwise 95% t intervals conditional on fixed lists. The internal-filtered interval includes zero. Table 1 reports all six training pools.

Limitations and Future Work

The authors acknowledge several limitations:

  • Pipeline Specificity: The audit primarily focused on a DINOv2-L/14 backbone; results might vary with different architectures.
  • Missing Metadata: The lack of cross-recording patient identifiers in the public CV2024 data means they could not definitively quantify patient-level overlap, which is the "gold standard" for medical leakage.
  • Screening vs. Detection: Their similarity screens are intended as auditing tools, not perfectly calibrated duplicate detectors.

Future work could involve applying ReuScope to a wider variety of model families and advocating for better metadata standards in public medical imaging benchmarks to allow for true patient-level auditing.

Summary updated

Abstract

We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study. It combines measurements of data relationships with two comparisons: changing the evaluation population while holding predictions fixed, and changing training data while holding evaluation fixed. Byte-identical files cross the public training-validation boundary, and all 41 Kvasir-Capsule recordings that contribute frames occur in both partitions. Shared source recordings establish cross-split overlap; missing cross-recording patient identifiers prevent counting unique shared patients. Across ten paired seeds, removing all training rows attributed to Kvasir-Capsule, plus five rows from other sources flagged as near-matches to it, lowers public-validation balanced accuracy from 82.5% to 61.2%. Evaluating the same fitted models on the validation rows of the other sources reverses the comparison: removal improves balanced accuracy by 1.38 percentage points. The public-validation drop measures sensitivity to source removal, not an amount of leakage-induced inflation. Matched controls show that sample size and class counts alone do not explain the observed score differences. An exploratory experiment removes the training neighbors of validation frames that have close training matches, or equally many distant training rows, and yields a difference-in-differences estimate, relative to matched unflagged frames, of -0.355 percentage points in true-class probability; the paired-t interval for accuracy spans zero. The resulting procedure distinguishes observed reuse from conditional performance sensitivity and specifies the population, class support and uncertainty unit for each conclusion.
BibTeX
@misc{choi2026reuscope,
  author = {Eunseob Choi and Kyeonghun Kim and Hyuk-Jae Lee and Nam-Joon Kim},
  title = {{ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024}},
  year = {2026},
  eprint = {2610.41835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41835},
}
Figure