Transferring Kernel Choice to Ten-Subject Clinical Prediction

When a clinical study has ten or twenty patients, deep networks overfit. The method that still works is often kernel ridge regression (KRR), a closed-form predictor that scores a new case by a weighted similarity to the training cases. KRR's quality then depends on one decision: the kernel, meaning the similarity function and its few settings (how fast similarity decays with distance, how smooth the fitted function is, how strongly it is regularized). With ten samples, that decision cannot be made reliably from the target data. Leave-one-out estimates on ten points are too noisy to tell a Gaussian from a Laplacian kernel, or one length scale from another. This proposal makes the decision before the target data arrives. We learn which kernel family and which hyperparameters work for small samples by simulating many ten-subject studies inside large source cohorts, and then we apply that frozen kernel to the real small study.

The opportunity sits between two bodies of work. Work on kernel generalization has built kernels beyond the default Gaussian: heavy-tailed Laplacian kernels, kernels with tunable smoothness (the Matérn family), and kernels that are mathematically valid on curved spaces such as the space of covariance matrices. Work on small clinical samples has shown that KRR beats fine-tuning below fifty subjects, but it always uses one fixed, standard kernel. Nobody has asked whether the wider kernel menu helps at N=10, or how one could choose from that menu when the target data is too small to choose with. Our core claim is that, below about fifty samples, most of the error that remains in kernel-based clinical prediction comes from the kernel and its hyperparameters being chosen badly. If that is right, transferring a low-dimensional kernel choice from large cohorts should recover much of the gap. This should hold even when the target trait is unrelated to the source traits, because the transferred kernel describes the geometry of the inputs rather than the meaning of any particular label.

Core Claim and What Is New

The motivating evidence comes from Multilayer meta-matching. It predicts behavioral traits from resting-state functional connectivity, the matrix of correlations between brain regions' activity at rest. It reports that classical KRR beats fine-tuned deep networks below 50 participants, and that stacking source-trained models gives useful predictions at 10 participants. That work has two limits. Its target-stage KRR treats connectivity matrices as flat vectors under a standard kernel. Its gains depend on the target trait correlating with source traits, and they decay when the correlation is low. We keep its finding that KRR is the right target model. We change what gets transferred: instead of trait predictions, we transfer the kernel.

Concretely, let $\theta$ collect a kernel family index, a decay or tail parameter, a smoothness parameter $\nu$, a length scale, and the ridge penalty $\lambda$. Let $\mathcal{S}$ be a collection of source prediction tasks, and let $D_n$ be a random $n$-sample training set drawn from a task $t$. We choose

$$\hat\theta_n = \arg\min_\theta \; \mathbb{E}_{t \sim \mathcal{S}} \, \mathbb{E}_{D_n \sim t} \big[ \mathcal{L}_t(\hat f_{\theta, D_n}) \big],$$

where $\hat f_{\theta,D_n}$ is the closed-form KRR fit and $\mathcal{L}_t$ is its error on held-out source subjects. Because the KRR solution is differentiable in $\theta$, the continuous parts of $\theta$ can be optimized by gradient descent through the closed form. We borrow this mechanism from Dataset Meta-Learning from Kernel Ridge-Regression, which differentiates through KRR to optimize synthetic support points. We use the same trick to optimize a handful of kernel parameters, so the memory cost that limits that work does not arise. The discrete family index is chosen by comparing the optimized objective across families.

The new element is the subscript $n$. The optimal kernel for ten samples need not match the optimal kernel for a thousand. Heavier tails and stronger smoothing may pay off precisely when data is scarce. We will measure how $\hat\theta_n$ shifts with $n$, not assume it is constant.

The Kernel Menu

The family deliberately includes the non-Gaussian and geometry-aware kernels from the first direction.

Heavy tails come from LaplacianFormer. It argues that Gaussian similarity over-suppresses intermediate distances when distances are heavy-tailed, and that an L1 Laplacian kernel preserves those correlations. That argument was made for attention in vision transformers trained on ImageNet. We test whether it carries over to noisy, high-dimensional clinical features at tiny $n$.

Geometry comes from treating connectivity and EEG covariance matrices as points on the curved space of symmetric positive definite (SPD) matrices. Two papers constrain the design here. Matérn Gaussian Processes on Riemannian Manifolds shows that plugging geodesic distance into a Matérn kernel generally breaks positive definiteness, and it builds valid kernels from the manifold's Laplacian spectrum, but only on compact manifolds, which excludes the SPD space. Invariant Kernels on Riemannian Symmetric Spaces proves that the geodesic Gaussian kernel is never positive definite on Riemannian symmetric spaces, and it gives a harmonic-analytic route to valid invariant kernels, such as the hyperbolic secant kernel on the hyperbolic plane.

These results rule out the naive geodesic Gaussian. We therefore include two kinds of geometric kernel. The first applies Gaussian, Laplacian, and Matérn kernels after a global matrix-logarithm map, which is valid because it is a Euclidean embedding. The second is an invariant SPD kernel built by the Da Costa et al. construction, included only where that construction gives something computable in our matrix sizes. We do not claim the second route will be tractable for 100-region connectivity matrices. It is the riskiest part of the menu and is treated as optional.

Research Program

The work has three parts. First, we build the episodic kernel-selection procedure over the menu above and run it on large source cohorts, yielding $\hat\theta_n$ for $n \in \{10, 20, 50, 100, 200\}$. Second, we analyze what transfers. We ask which family wins at each $n$, whether the selected length scale and penalty drift systematically as $n$ shrinks, and whether the selection is stable across disjoint subsets of source tasks. Third, we test the frozen kernels on held-out target cohorts and traits. The central comparison is against choosing the kernel on the target itself, and against meta-matching, which transfers predictions rather than the kernel. Because our kernel replaces only the target-stage KRR, it can also be inserted inside meta-matching. We will report that combination too.

Hypothesis Sketching for Online Kernel Selection also selects kernels in a continuous space. It does so online on large data streams (cod-rna) with regret guarantees. We select offline from source episodes because the target stream here has ten points.

Baselines

The naive reference point is target-only KRR with the default kernel used by meta-matching, with the ridge penalty tuned by leave-one-out on the target samples alone. A second naive variant runs the same nested search over our full kernel menu but uses only the target data. This variant tells us whether a richer menu without transfer helps or simply overfits the choice. Both get identical folds and the same grid density as our method.

The strongest existing method for connectivity-based trait prediction is multilayer meta-matching (Multilayer meta-matching). We will run it with its released configuration on the same source cohorts (UK Biobank and the other four sources) and the same HCP-YA and HCP-Aging target splits. No reported result was found on Lacuna for meta-matching's numerical prediction accuracy at 10 or 50 participants, so we will regenerate its numbers rather than quote them. Beating it means a higher mean Pearson correlation between predicted and true trait values at $n=10$ and $n=20$, averaged over many random target draws, with a paired test across traits. Matching it on traits that are well correlated with the source traits, while beating it on traits that are poorly correlated, would also count as a meaningful result.

On the EEG side, the strongest competitor is Adaptive Split-MMD (Adaptive Split-MMD Training). It aligns source and target classifier outputs with a Gaussian maximum mean discrepancy penalty, a measure of distribution difference computed through a kernel. With 10 target trials per subject it reports accuracy 0.66 and AUC 0.74 on Active Visual Oddball (target-only: 0.52 and 0.53), and accuracy 0.61 and AUC 0.65 on ERP CORE P3. We will rerun it under its protocol: same five electrodes, 80 source trials and 10 target trials per subject. Beating it means a higher AUC than our rerun on both datasets. We will also report the published figures for reference. A secondary EEG competitor is TSMNet (SPD domain-specific batch normalization). It aligns covariance matrices on the SPD manifold before a tangent-space linear classifier. No reported number was found on Lacuna for TSMNet; we will run it from its released code on BNCI2014001 and BNCI2015001, subsampled to few-shot target sessions.

Deep Kernel Transfer meta-learns a neural feature map that defines a kernel. It appears in the file as the baseline that Non-Gaussian Gaussian Processes for Few-Shot Regression beats on Power (negative log-likelihood 1.35 versus 0.22 for the non-Gaussian model). It is the closest method that also transfers a kernel. We will train it on the same source episodes with a matched number of meta-training steps.

Two ablations isolate our ingredients. The first transfers hyperparameters for a fixed Gaussian kernel only. It asks whether the gain comes from transfer alone, with the wider kernel family adding nothing. The second fixes the family to log-Euclidean Gaussian with transferred hyperparameters. It asks whether geometry alone is enough. All methods share the same source data, the same target draws, and a fixed budget of tuning evaluations.

Why not just…?

Why not just fine-tune a pretrained network on the small cohort? Meta-matching directly found that fine-tuned deep networks lose to plain KRR below 50 participants. Unveiling COVID-19 from Chest X-ray shows that high-capacity models trained on small cohorts latch onto scanner and site artifacts. We include fine-tuning in the $n$ sweep to confirm the crossover in our splits, but we do not expect it to compete at $n \le 20$.

Why not just tune the kernel by cross-validation on the target data? This is the most natural alternative, and the episodic setup lets us measure its failure directly. Inside source cohorts we know the true best kernel for each simulated ten-subject study, so we can compute how often target-only leave-one-out picks it and what accuracy is lost when it does not. If target-only selection over the full menu performs as well as the transferred choice, our central claim is false and we will report that.

Why not just project to the tangent space and use a Gaussian kernel there? This is a real threat. The log-Euclidean Gaussian is valid, cheap, and close to what TSMNet does before its classifier. It may capture all the geometric benefit, leaving the invariant kernels with nothing to add. The second ablation settles this question. If it ties the full menu, the conclusion is that geometry matters only through the log map and that the remaining gain comes from transferred tails and smoothing. That would be a useful and simpler recommendation.

Why not just meta-learn a deep kernel, as Deep Kernel Transfer does? A learned feature map has many parameters tied to the source labels. We expect it to transfer poorly when the target trait differs from the source traits, which is exactly where meta-matching also weakens. Our kernel has about five parameters, which should make it more robust there. The low-correlation trait split tests this directly. If Deep Kernel Transfer wins on that split, the low-dimensional restriction is not paying for itself.

Why not just add synthetic patients with a diffusion model? DiffMDD reports gains on MDD EEG, but Limitations of Synthetic Data Generation finds that diffusion augmentation rarely beats simple augmentations in specialized sparse domains. A generator trained on ten subjects has the same data problem as the classifier. We therefore treat augmentation as complementary and do not run it as a primary baseline.

Proposed Experiments

The primary testbed is connectivity-based trait regression. We use the five source cohorts of meta-matching for episode simulation, and we hold out HCP-YA and HCP-Aging as targets. Targets are split into traits highly correlated with source traits and traits weakly correlated with them. Each method is run on 100 random target draws at each $n$.

The secondary testbed is P300 classification on Active Visual Oddball and ERP CORE P3 under the Split-MMD protocol. Here we use trial covariance matrices with kernel ridge classification. We also evaluate few-shot motor imagery on BNCI datasets.

Three results decide the proposal. First, in source episodes, the gap between target-only kernel choice and the best available kernel must be large at $n=10$ and must shrink toward zero by $n=200$. If it is already small at $n=10$, kernel selection is not the bottleneck and the project stops there. Second, the transferred kernel must beat target-only tuning and the fixed-Gaussian transfer ablation on held-out traits at $n \le 20$. Its advantage over meta-matching should be largest on the weakly correlated traits, which is the regime meta-matching itself flags as weak. Third, the selected kernel parameters should move systematically with $n$, for example toward heavier tails and stronger penalties at small $n$. A consistent drift would show that "the right kernel" depends on sample size, which is a finding about kernels themselves that neither literature currently states.