Rank Inflation as an Audit of Undocumented Data Splits

When a model's size is tuned on a validation set, such as how many latent factors a matrix factorization keeps or how many principal components feed a classifier, the chosen size is only as trustworthy as the independence of that validation set. If the train and validation splits secretly share structure, a model can look better with more factors because it keeps fitting that shared structure. Examples are slides from the same patient, ratings from the same session, or cell lines from the same tissue. Papers rarely document how splits were made. This proposal treats the selected dimensionality itself as evidence about the data pipeline. We re-run automatic dimensionality selection under a split built from groups the method infers on its own, and measure how much the chosen dimension and the reported score shrink. The result is a quantitative, label-free check on whether a reported result depends on undocumented split dependence. It needs no metadata and no raw-data release.

The motivation is concrete. Reproducibility of deep learning in digital pathology whole slide image analysis reimplemented three Camelyon challenge methods from their papers alone. Slide-level accuracy for one method fell from 92% to 54.7%, and lesion detection score (FROC) for another fell from 0.981 to 0.499. The authors traced much of this to unreported preprocessing and patient-level splitting. Their remedy is a 12-point checklist, which depends on authors volunteering exactly the information that was missing. Meanwhile, adaptive rank selectors such as the block-growing randomized PCA in Efficient Model-Based Collaborative Filtering with Fast Adaptive PCA stop adding factors when validation error bottoms out. They have no defence if that validation set is contaminated. Nobody has measured how validation-driven dimensionality selection fails under controlled, undocumented split dependence, or turned that failure into a diagnostic.

Core Claim

Hidden group structure that straddles a split shows up in a specific way. Some latent factors have sample scores concentrated on clusters of samples, and these factors improve validation error only when the clusters appear on both sides of the split. Our audit fits a factor model and then flags factors whose scores are cluster-concentrated. It builds a pseudo-grouped split by clustering samples on those factors only, and re-runs the same validation-driven rank selection.

We report two quantities. The first is the rank gap $\Delta K = K_{\text{rand}} - K_{\text{grp}}$, where $K_{\text{rand}}$ is the rank chosen under the original random split and $K_{\text{grp}}$ is the rank chosen under the inferred-group split. The second is the score gap $\Delta s$, the drop in the downstream metric between the two splits. We claim three things. Both gaps are near zero when no group structure crosses the split. Both grow with the fraction of leaked groups. The specific factors that disappear name the nuisance structure, for example patient-specific staining or a hospital scanner, and that can be written into a dataset's documentation as a number rather than a sentence.

A second reference point is a validation-free rank estimate. We use FDR-controlled tile selection from The Trustworthy Pal: Controlling the False Discovery Rate in Boolean Matrix Factorization for binary data, and the posterior number of active components in GFA: Exploratory Analysis of Multiple Data Sources with Group Factor Analysis for continuous multi-view data. These estimate how much structure is statistically real in the training data, nuisance groups included. So the expected ordering under leakage is that the validation-free rank is close to $K_{\text{rand}}$ and both exceed $K_{\text{grp}}$. A clean split should instead give $K_{\text{rand}} \approx K_{\text{grp}}$, with both no larger than the validation-free rank. TrustPal's guarantees assume independent Bernoulli noise, and group structure violates that assumption. We use it as a reference count of significant structure, not as a leakage test.

Research Program

The first component is a leakage simulator applied to real data that has hidden group labels. On Camelyon17, patient and center identities exist. We hide them from the method and use them only to construct splits with a controlled fraction of patients straddling train and validation, from fully grouped to fully random. On the Genomics of Drug Sensitivity in Cancer (GDSC) multi-omics data used by GFA, tissue of origin plays the same role. On MovieLens-1M and MovieLens-20M, the analogue is temporal: random rating splits against time-ordered splits, plus injected synthetic user clusters. We also generate synthetic Boolean and Gaussian low-rank matrices with planted group-specific factors, where the true transferable rank is known.

The second component is the audit itself. It fits a factor model, scores each factor by how cluster-concentrated its sample scores are, clusters samples on the flagged factors, and re-runs the original selector under the resulting grouped split. For Camelyon we work on patch embeddings from a fixed pretrained network. There, "dimensionality" is the number of principal components passed to a linear probe, grown in blocks until validation accuracy stops improving, mirroring the Fast Adaptive PCA stopping rule.

The third component is the failure envelope promised by Gap 3 of the file. We plot $\Delta K$ and $\Delta s$ against leakage fraction for each selector: Fast Adaptive PCA, GFA, TrustPal, and plain cross-validated rank grids. This shows which selectors inflate most. That result is useful even if the audit's group inference turns out to be weak.

Baselines

The naive reference point is to report what the original pipeline reports: random-split validation, rank chosen by the stopping rule, and score taken at face value. Every gap is measured relative to it.

The oracle is grouped cross-validation using the true hidden group identities: patients on Camelyon17, tissue type on GDSC, time on MovieLens. It is the strongest possible competitor because it is exactly what a fully transparent dataset would let a practitioner do. Our audit beats the naive baseline if its $\Delta s$ recovers at least half of the oracle's score drop. It earns its keep if it gets close to the oracle without seeing the metadata. No reported result found on Lacuna for grouped-versus-random split gaps on these benchmarks. The closest number is the reproduction drop in Reproducibility of deep learning in digital pathology whole slide image analysis, slide accuracy 92% to 54.7% for the Lee algorithm, and that drop mixes split leakage with preprocessing changes.

The strongest metadata-free competitor is grouped cross-validation on clusters of the raw inputs or full embeddings, for example k-means with the cluster count matched to ours. It gets the same embeddings, the same selector, and the same number of cluster settings tried. Our method beats it only if it recovers more of the oracle gap at equal tuning budget.

A near-duplicate baseline flags validation samples whose nearest training neighbour falls below a distance threshold, with the threshold tuned on synthetic data. It is evaluated by the area under the curve (AUC) for predicting which runs are leaky.

The key ablation clusters on the factors that were not flagged, or on random subsets of factors of the same size. It isolates whether the cluster-concentration score is doing the work.

Why not just…?

Why not just require authors to document group IDs and use grouped cross-validation? When the metadata exists and is released, that is the right answer, and it serves as our oracle. The point of Reproducibility of deep learning in digital pathology whole slide image analysis is that such details are routinely missing even from peer-reviewed top-performing work. Clinical and commercial data often cannot be released at all. The audit targets exactly the case where the oracle is unavailable.

Why not just cluster the raw data and split by cluster? This is the real threat. Our argument is that raw clustering is dominated by the largest sources of variance, such as tissue type or overall stain intensity, rather than the factors that only help when groups cross the split. That argument is a hypothesis, and the head-to-head comparison settles it. If raw clustering recovers the oracle gap as well as factor-restricted clustering, we drop the claim about the audit statistic. The contribution then reduces to the failure-envelope benchmark of validation-driven selectors, which stands on its own.

Why not just detect near-duplicates between splits? Duplicate checks catch copied or near-identical samples. Same-patient patches from different tissue regions, or different ratings by the same user, are not near-duplicates, yet they share a latent factor. The synthetic setting with planted group factors and no duplicates tests this directly. If the duplicate detector matches us there, this objection wins.

Why not just compare intrinsic dimension between splits? Global estimators such as the nearest-neighbour MLE in Maximum Likelihood Estimation of Intrinsic Dimension and TwoNN, which Intrinsic dimension of data representations in deep neural networks used to show that random-label training keeps final-layer intrinsic dimension high, summarise the geometry of the whole data. Leaking groups across a split does not change that geometry, so these estimators should be blind to it. We include them as a negative control. The MLE is also known to be biased downward in high dimensions, which further limits it as an audit tool.

Proposed Experiments

The decisive check is on Camelyon17 with patient identities hidden. As the fraction of straddling patients rises from 0 to 1, $\Delta K$ and $\Delta s$ should rise monotonically. They should stay near zero at fraction 0, and they should track the oracle grouped-split gap better than raw-cluster grouping and the duplicate detector, measured by rank correlation across leakage levels and AUC for flagging leaky runs.

On GDSC we check two further things. GFA's posterior component count should sit near $K_{\text{rand}}$ under tissue leakage; GFA's reported predictive performance peaks at 20–25 components. The factors that vanish under our grouped split should align with tissue type. On MovieLens the audit should separate random from temporal splits. On synthetic data with known transferable rank, we report how closely $K_{\text{grp}}$ recovers the true transferable rank as noise and leakage fraction vary.

The proposal fails if any of three things happens: factor-restricted clustering does no better than raw clustering, $\Delta K$ is non-zero on clean grouped splits, or the flagged factors show no alignment with the hidden groups.