Does Feature Learning Collapse the Kernel? Distance Floors for Deep Kernel Machines

Deep kernel machines are a family of models that learn image features the way a convolutional network does, while keeping the probabilistic bookkeeping of a Gaussian process (a model that places a prior over functions and reports how unsure it is). The best of them now classifies CIFAR-10 at about 92.7% accuracy. That makes them the first purely kernel-based method that competes with tuned convolutional networks. There is a catch. As these models are given more capacity, their accuracy rises but their test log-likelihood gets worse. In other words, they become confidently wrong more often. We propose that this is not a generic overfitting problem. It is a specific geometric failure that hybrid neural-network-plus-GP models are already known to have: the learned representation pulls unfamiliar inputs onto familiar ones, so the model can no longer tell them apart.

The note does two things. First, it tests that diagnosis directly by measuring, layer by layer, how much the learned kernels shrink distances between test or out-of-distribution images and the training set. Second, it proposes a minimal fix that operates in kernel space. Each layer's learned similarity matrix is written as a fixed fraction of the untrained prior kernel plus a freely learned positive part. This puts a hard floor under how far distances can shrink at that layer, while leaving the model free to stretch distances wherever that helps classification. If the diagnosis is right, the floor should recover calibrated likelihoods and out-of-distribution detection at a small cost in accuracy. If it is wrong, the measurements will show it, and that negative result is informative in its own right.

Core Claim

Convolutional Deep Kernel Machines extends deep kernel machines to images. It replaces spatial inducing points (a small learned set of summary inputs that keeps cost manageable) with non-spatial inter-domain ones, and it adds kernel versions of batch normalisation and global average pooling. It reports 92.69% on CIFAR-10 and 72.05% on CIFAR-100. The same paper also reports the pathology that motivates this proposal: test log-likelihood degrades as the number of inducing points grows, which the authors describe as the model overfitting its own uncertainty.

Separately, On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty shows a related failure in models that put a GP on top of a neural feature extractor. The training objective rewards the extractor for mapping distinct, and even out-of-distribution, inputs into the same tight region, and the GP then treats them as familiar. That paper fixes the problem by making the network bi-Lipschitz, meaning distances are neither blown up nor crushed. It does so with spectral normalisation and residual connections.

Nobody in the file has connected these two observations. Our claim is that they are the same phenomenon. The layer-wise Gram matrices a deep kernel machine learns are its features. Optimising them for fit contracts the directions in which test and out-of-distribution points differ from training points. And this contraction tracks the log-likelihood degradation as inducing points increase.

A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs gives theoretical reason to expect this. It shows that feature learning in finite convolutional networks appears as outlier eigenvalues aligned with the targets. The learned kernel concentrates on label-relevant directions, which implies that other directions shrink.

The fix uses the structure of the deep kernel machine objective from A theory of representation learning gives a deep generalisation of kernel methods. There, each layer $l$ has a learned Gram matrix $G_l$ over inducing points, regularised toward $K(G_{l-1})$, the kernel that the untrained infinite-width prior would produce from the previous layer. We reparameterise

$$G_l = \alpha\, K(G_{l-1}) + L_l L_l^\top, \qquad 0 < \alpha \le 1,$$

where $L_l$ is a freely learned matrix and $\alpha$ is the floor fraction. Squared feature distance under a Gram matrix is $d_l^2(x,x') = G_l(x,x) + G_l(x',x') - 2G_l(x,x')$. Distances add across a sum of positive kernels, so this construction guarantees

$$d_l^2(x,x') \ge \alpha\, d_{K}^2(x,x'),$$

where $d_K$ is the distance under the prior map. In words, a layer can never shrink a distance below a fixed fraction of what the prior would assign, but it can enlarge distances without limit.

This is a one-sided, kernel-space analogue of the bi-Lipschitz constraint. It is deliberately different from it. Spectral normalisation in DUE bounds expansion as well as contraction, and that is the expressivity cost the DUE paper itself acknowledges. We only bound contraction.

Research Program

The work has three stages.

Stage 1: diagnosis. We train Convolutional DKMs at several inducing-point counts, using the published recipe from Milsom et al. For each layer we compute a collapse ratio: the learned-kernel distance from a test image (or an out-of-distribution SVHN image) to its nearest training image, divided by the same distance under the prior kernel chain. We then ask whether the median ratio for out-of-distribution points falls as inducing points increase, and whether it predicts test log-likelihood across runs. The theory from Naveh and Ringel suggests a further check: whether the contraction is concentrated outside the top eigen-directions of the label kernel.

Stage 2: the floor. We implement the floor parameterisation inside the sparse DKM objective. One technical question has to be answered first. Predictions at test points pass through a conditional formula extended from the inducing points, and the guarantee only carries to test inputs if the prior term and the learned term are extended separately, each as a valid positive kernel. We will derive and verify that extension before running any large experiment. If it fails, the guarantee holds only on inducing points, and we will report that limitation.

Stage 3: transfer to a stochastic deep GP. As a smaller check, we repeat the collapse-ratio measurement on a 2–3 layer Deep convolutional Gaussian process. That model propagates samples rather than Gram matrices, and it reported no rank collapse but never tested out-of-distribution inputs. Whether stochastic layers resist distance collapse better than deterministic Gram optimisation is an open question in its own right.

Baselines

Unmodified Conv-DKM. The naive reference point is the Convolutional DKM itself, trained with the published architecture and inducing-point schedule. It is the strongest pure-kernel result in the file: 92.69% CIFAR-10 accuracy and 72.05% CIFAR-100 accuracy (Convolutional Deep Kernel Machines). The file gives no numeric test log-likelihood or calibration error for it, so we found no reported result on Lacuna for Conv-DKM log-likelihood, and we will measure it ourselves. The floored model uses identical data augmentation, inducing counts and GPU budget. Beating this baseline means a higher test log-likelihood at the largest inducing count, and no downward log-likelihood trend as inducing points grow, with accuracy within one point.

Strongest single-pass uncertainty competitor. On out-of-distribution detection, the strongest competitor is DUE (On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty), a spectrally normalised residual network topped by an inducing-point GP. It reports 0.958 AUROC on CIFAR-10 versus SVHN, against 0.940 for SNGP and 0.967 for a five-model deep ensemble. We will retrain DUE with our evaluation code rather than reuse its number, so that splits and preprocessing match. Matching or exceeding 0.958 with a pure kernel model would close the gap. We do not expect to beat the ensemble's 0.967, and we will report the comparison either way.

Calibration reference. For calibration we use Guided Deep Kernel Learning, which uses an infinite-width network GP as a teacher during training. It reports 78.36% accuracy and 0.06 expected calibration error on CIFAR-100, versus 77.90% and 0.08 for plain deep kernel learning. Its accuracy is higher than any pure kernel model's, so the fair comparison is calibration error at whatever accuracy each method reaches. We do not claim an accuracy win here.

Key ablation. The ablation that isolates our ingredient is a two-sided version of the floor, which also caps how far distances can grow. If the two-sided version calibrates equally well but loses accuracy, that supports the claim that bounding only contraction is what matters.

Why not just…?

Why not just turn up the KL regulariser that already ties each layer to the prior? This is the most serious threat, and we treat it as a co-equal hypothesis rather than a strawman. The deep kernel machine objective already penalises how far $G_l$ moves from $K(G_{l-1})$, so a heavier penalty also discourages collapse. The difference is that the penalty taxes every departure from the prior, including useful expansions, and it does so on average over inducing points. The floor taxes nothing above a hard contraction bound. We will sweep the regulariser weight under the same tuning budget as $\alpha$, and plot log-likelihood against accuracy for both families. If the regulariser sweep traces the same frontier, the floor adds nothing and the contribution reduces to the Stage 1 diagnosis.

Why not just use fewer inducing points, or stop early? Milsom et al.'s own observation is that fewer inducing points give better likelihood and worse accuracy, so this just picks a point on the existing trade-off curve. We will include it on the same frontier plot. The floor is worthwhile only if it dominates that curve.

Why not just apply temperature scaling after training? Rescaling the output can repair in-distribution log-likelihood, but if distances to SVHN images have collapsed, no rescaling restores their separation from training images. We will apply temperature scaling to every method. The decisive comparison is out-of-distribution AUROC and predictive variance, which scaling cannot change in rank order.

Why not just use DUE or a deep ensemble and abandon pure kernel methods? For practitioners who only need out-of-distribution detection, that may be the right answer, and the file's numbers favour the ensemble. The question here is whether a fully kernel-based feature learner can be accurate and calibrated at once. If the floored model reaches DUE's AUROC while keeping Conv-DKM's accuracy, the answer is yes. If it cannot, the answer is no, and we will say so.

Proposed Experiments

The datasets are MNIST for fast sweeps, CIFAR-10 and CIFAR-100 for the main results, and SVHN as the out-of-distribution set, which matches the protocol in DUE. The metrics are accuracy, test log-likelihood, expected calibration error, out-of-distribution AUROC, and the per-layer collapse ratio. Full-size Conv-DKMs cost about 77 A100 hours each, so all sweeps over $\alpha$, regulariser weight and inducing count use a reduced-width model. Only the final configurations are trained at full size.

Two checks decide the outcome.

First check: does collapse explain the degradation? In unmodified Conv-DKMs, the out-of-distribution collapse ratio should fall monotonically with inducing count and should correlate with the log-likelihood degradation across runs. If it does not, the diagnosis is wrong and Stage 2 is abandoned.

Second check: does the floor beat the obvious knob? On the log-likelihood-versus-accuracy plane, the floored models should sit strictly above the curve traced by the regulariser sweep and the inducing-count sweep. They should also raise out-of-distribution AUROC above unmodified Conv-DKM, and close to DUE's 0.958.

Secondary check: do stochastic layers behave differently? The DeepCGP measurement tells us whether sampling-based stochastic layers show the same contraction pattern. That result shapes whether the floor idea should be carried over to stochastic deep GPs at all.