Honest p-values for Similarity Network Fusion subtypes in small cohorts

Similarity network fusion (SNF) is a popular way to find patient subtypes when each patient has been measured several ways, for example brain scans plus clinical questionnaires, or gene expression plus methylation. Each measurement type is first turned into a patient-by-patient similarity graph. The graphs are then merged by repeatedly passing information through each patient's nearest neighbours, and the merged graph is cut into clusters. Analysts then ask whether those clusters differ in diagnosis or survival. In small cohorts this last step is badly miscalibrated. The clusters, the tuning choices (neighbourhood size, kernel width, which data types to include), and the outcome test all use the same few dozen patients. Tools such as metasnf explicitly pick among hundreds of candidate clusterings partly by clinical usefulness. The p-value reported for the winning solution therefore has no known error rate.

We propose to borrow the central trick of the network-based statistic (NBS), a brain-connectivity method that controls false positives over thousands of edges by comparing the largest observed effect against a permutation distribution of the largest effect. We apply that trick to the whole family of SNF solutions an analyst might plausibly pick. We add a second permutation null that asks a question SNF currently cannot answer: whether the fused structure is supported jointly by the different data types, or only by something they share for uninteresting reasons, such as acquisition site. The deliverable is a test with controlled family-wise error. It will be judged by whether its p-values predict which small-cohort subtype findings replicate in independent patients.

Core Claim

Our first claim is that the right unit of inference for SNF in small cohorts is the whole search over solutions, not the selected solution. Let $h$ index a grid of fusion settings: neighbourhood size $k$, kernel scale $\alpha$, number of clusters, and data-type subsets. Let $T_h$ be the association statistic between the clusters under setting $h$ and the outcome, such as a Kruskal–Wallis or log-rank statistic. The fused graphs never see the outcome, so the graphs and their clusterings can be computed once and cached. Permuting the outcome $B$ times then only requires recomputing $T_h$. This gives a max-statistic null:

$$p_h = \frac{1 + \#\{b : \max_{h'} T^{(b)}_{h'} \ge T_h\}}{1 + B},$$

where $T^{(b)}_{h'}$ is the statistic for setting $h'$ under the $b$-th outcome permutation. This controls the family-wise error rate across every solution in the search for the same reason NBS's maximal-component null does. It also turns metasnf-style exploration from a source of bias into a source of power.

Our second claim is that cross-diffusion amplifies whatever the data types share, and in multi-site small cohorts the thing they share is often site or batch. The noise-suppression argument for SNF says that edges without local support in the other data type are damped. The argument says nothing about artifacts that are present in both data types. We therefore propose a structural null. Patient identities in one data type are shuffled relative to the others, but only within site strata. SNF is then rerun, and the resulting agreement statistic (the modularity of the fused graph) is compared to its observed value. A fused structure that survives this null is supported by cross-modal agreement beyond site.

The outcome permutation is likewise stratified by site. This connects to Two Sample Inference for Populations of Graphical Models, which showed that treating estimated connectivity graphs as if they were directly observed inflates false discoveries. Our view is that SNF inherits and can amplify that problem, and that stratified permutation is the minimal fix at the level of the fused patient graph.

Research Program

The method has three parts. The first is the cached max-statistic outcome test over the SNF solution grid, computed as described in the Core Claim. The second is the within-site view-shuffle null for fused structure, which is the expensive part because SNF must be rerun for each permutation. We will budget it at a few hundred permutations and report Monte Carlo error.

The third part is an inductive assignment rule, needed so that replication can be measured without leakage. The fused graph and clusters are frozen on the discovery cohort. A new patient is assigned to a cluster using only that patient's per-data-type affinities to discovery patients, propagated through the frozen fused graph. No diffusion is recomputed over the combined cohort. This responds directly to the concern that label propagation on held-out splits, as in metasnf, lets test patients shape the training graph.

We will also study the grid itself as the analogue of NBS's arbitrary primary threshold. Because the test takes the maximum over the grid, a larger grid costs some power. We will measure how quickly power decays as the grid grows. We will also measure whether a coarse, pre-registered grid loses much relative to a fine one.

Prior work positions the proposal as follows. DeepMoIC and DeepASD build fused patient graphs and train graph neural networks for classification. They report accuracy but make no inferential claim and do no cross-cohort test, so they supply the fusion machinery but not the question we ask. metasnf motivates the work: it builds exactly the solution family we test over, but it assesses solutions by stability and utility rather than calibrated error. MOAT supplies the closest methodological precedent. It uses network-level permutation inference with family-wise error control, but for structure-to-function association at biobank scale ($N=4{,}242$), and without patient-similarity fusion. SIMMS and GSPLS constrain the proposal. They show that structure imposed from biological databases helps small-sample generalization, so our method must beat an SNF analysis that is simply validated honestly, not an SNF analysis done carelessly.

Baselines

The naive reference point is current practice. The analyst runs SNF across the grid and picks the solution with the strongest outcome association, or the metasnf representative judged most useful. The analyst then reports that solution's unadjusted p-value. We expect this to have grossly inflated type I error in small cohorts. Showing how inflated it is forms part of the contribution.

The strongest honest competitor is sample splitting with external confirmation. The search and selection are done on half the cohort. The chosen solution is frozen and assigned inductively to the other half, where a single test is performed. This is valid by construction and is what a careful reviewer would demand. It runs on the same data, the same grid and the same inductive assignment, so the only difference is how the data are used. Beating it means equal or better power and replication rate at matched nominal error. We expect this to be hardest at $n$ between 40 and 100.

The strongest existing method for fused-structure assessment is metasnf's stability analysis, which uses the agreement between solutions across perturbations as measured by the Adjusted Rand Index. We will run it with its default perturbation settings on the same cohorts. We will ask whether high stability separates structure driven by site from structure that crosses data types; our method beats it if our view-shuffle null does and stability does not.

Two ablations isolate the key ingredients. The first replaces cross-diffusion with the arithmetic mean of the affinity matrices, as in Deng et al., and keeps the identical max-statistic test. This tells us whether the nonlinear fusion matters once inference is honest. The second removes site stratification from both permutation nulls, to measure how much of the apparent structure and power is site. On reported numbers, we found no reported result on Lacuna for calibrated type I error or replication rate of SNF subtype–outcome tests. The only relevant inferential figures are edge-level: on synthetic graphs, R³ reports a true positive rate of 0.96 and a false discovery proportion of 0.15, against 0.25 and 0.74 for standard edge testing. We cite this as evidence of the size of the problem, not as a directly comparable baseline.

Why not just…?

Why not just hold out half the data? Splitting is valid and is our main competitor, so this is a real threat. Its cost is that selection and testing each see only half of an already tiny cohort. With 40 patients that often means neither half can support a subtype. The permutation approach uses all patients for both steps and pays for the search through the max statistic instead. Experiment 1 settles the question directly by comparing power at matched error across $n$. If splitting matches us below $n=60$, we will say so and recommend it there.

Why not just average the similarity matrices instead of cross-diffusing? Averaging keeps artifacts found in only one data type, while cross-diffusion damps them. The reverse may also hold: cross-diffusion may amplify artifacts shared across data types, such as site, more than averaging does. We do not assume either direction. The mean-fusion ablation, run under identical inference, measures it. If mean fusion replicates as often as SNF, the honest conclusion is that the inference layer, not the fusion rule, is the contribution.

Why not just concatenate features and fit LASSO or a random forest? That approach answers a different question: it predicts a known label rather than discovering subtypes. It is also known to let a high-dimensional data type drown a smaller informative one. Evidence from SIMMS, which beat single-feature learners across 11,392 patients, and from GSPLS, where mixOmics reached AUC 0.532 on GSE33356, suggests naive high-dimensional learners generalize poorly at small $n$. We will nonetheless include a concatenated-feature k-means subtype pipeline under the same max-statistic test, so the comparison is inference against inference.

Why not just use consensus-clustering stability? Stability measures whether a clustering reproduces under perturbation. Site effects are highly reproducible, so a stable clustering can still be an artifact. The site-recovery check below tests exactly this.

Proposed Experiments

Experiment 1 is a simulation to test error rates and power. We will simulate two or three data types with a planted subtype structure that crosses data types, plus a site effect shared across data types, at $n \in \{20, 40, 80, 150\}$ and three sites. Under no outcome effect, the naive pipeline's false-positive rate should far exceed 0.05, while ours should sit at 0.05 within Monte Carlo error. Under planted effects we will compare power against splitting and against the mean-fusion ablation.

Experiment 2 tests replication in TCGA multi-omics data. We will use the BRCA cohort ($N=511$; mRNA, copy number and methylation, as in DeepMoIC). From it we will repeatedly draw discovery subsamples of 20 to 100 patients, run each pipeline, and assign the remaining patients inductively to the frozen clusters. There we will test the same subtype–survival and subtype–label association. The primary metric is the replication rate among discoveries declared at family-wise error 0.05. The secondary metric is the correlation between discovery p-values and held-out effect sizes. The decisive check is whether our p-values rank findings by replicability better than naive p-values do.

Experiment 3 is a multi-site confound test on ABIDE. We will use functional connectivity and phenotypic data from several ABIDE sites. The small single-site anchor is UCLA Sample 1 ($n=73$), the cohort analysed by R³. We will evaluate held out one site at a time. The decisive check is whether naive SNF clusters recover acquisition site more than diagnosis, as measured by the Adjusted Rand Index against site labels. It is also whether the within-site view-shuffle null rejects such structure while metasnf stability does not. For context, DeepASD reports 87.38% accuracy on ABIDE A. That figure is a pooled classification result, not a leave-site-out inferential one, so we report it only as context.

Experiment 4 uses small GEO data. We will apply the pipeline to GSE33356 ($N=84$), with GSPLS's AUC of 0.807 as the reported reference for supervised prediction on that cohort. The purpose is to show what survives honest inference in a genuinely small multi-omics study.

The proposal fails if splitting matches our power and replication rate at every sample size. It also fails if the site-stratified null removes nearly all structure in real data. Either result would still be a useful negative finding about SNF subtyping in small cohorts.