Is Representation Rigidity Real? Separating Missing Features from Badly Shaped Ones

When a large pretrained image model is adapted to a specialized domain such as histology slides, retinal scans or chest X-rays, a small classifier trained on its frozen features often does poorly. Adding lightweight trainable modules, or opening more layers to training, recovers only part of the loss. The field's name for this is representation rigidity: the frozen model supposedly lacks the features the new domain needs. There is a second, cheaper explanation that nobody has separated from the first. The information may be present, but the features may be distributed so awkwardly (skewed, heavy-tailed, stretched along a few directions) that a probe fit on a handful of labeled examples cannot use it. If much of the "rigidity gap" is really this shape problem, it can be fixed in closed form on frozen features, with no backpropagation and no extra model capacity.

This note proposes to build the strongest frozen-feature classifier we can justify on statistical grounds, and to use it as a measuring instrument. The classifier first reshapes features toward a bell curve with a learned per-dimension power transform. It then applies a hierarchical Bayesian Gaussian classifier whose prior is learned across many related tasks. "Hierarchical Bayesian" here means that each new task's parameters are pulled toward values that worked on earlier tasks, and the pull weakens as that task's own data grows. We will measure how much of the gap between frozen probing and adapter fine-tuning this classifier closes on each domain. We will also test whether its own Bayesian evidence, computed from frozen features alone, predicts in advance which tasks genuinely need the backbone to change.

Core Claim

The claim has two parts. First, a substantial part of the gap usually attributed to rigidity is a feature-shape gap. We propose to close it with a transform whose parameters are shared hierarchically across tasks, because in the 1-to-16-shot regime a single task cannot estimate its own transform reliably. Second, the residual gap that remains after the shape correction is the true capacity deficit. That residual should be predictable from a held-out likelihood score computed by the Bayesian model before any fine-tuning is run. The second part is the more useful one. It turns a diagnosis of "rigidity," which today is only made after the fact by running expensive adaptation, into a cheap test.

What the Literature Leaves Open

Evidence for the shape explanation comes from work that never asked the rigidity question. Two papers transform features toward normality on frozen backbones and get large gains without touching backbone weights. Free Lunch for Few-shot Learning: Distribution Calibration applies Tukey's power transform (an element-wise $x^\lambda$ that pulls in long positive tails) and then borrows statistics from similar base classes. FeCAM combines the same square-root transform with class-specific covariance and shrinkage for continual learning. Both use a fixed exponent chosen by hand and learn nothing across tasks. On the Importance of Gaussianizing Representations shows that standard batch and layer normalization fix only the mean and variance while higher-order normality decays with depth. It fits a Yeo-Johnson transform (a power transform that handles negative values) inside the network with a one-step Newton update. We reuse that estimator, but on frozen features and with a prior over its parameter.

Evidence that rigidity is also partly real comes from the adaptation literature. That work is motivating and also constraining, because it shows fine-tuning sometimes helps.

All four add capacity, and none checks how much of their gain a well-shaped frozen probe would already have recovered.

The Bayesian side supplies the method. A Hierarchical Bayesian Model for Few-Shot Meta Learning (NIW-Meta) uses a Normal-Inverse-Wishart prior, the standard conjugate prior over a Gaussian's mean and covariance. Conjugacy makes task updates algebraic rather than iterative. However, NIW-Meta places the prior on network weights under a local quadratic approximation, and it assumes the incoming features are well behaved. LiFT places a mixture prior over adapter weights and samples it with Langevin dynamics. Its prior is also over parameters, not over the feature distribution, and it cannot create features the backbone lacks. Recasting Gradient-Based Meta-Learning as Hierarchical Bayes and Modular Meta-Learning with Shrinkage justify reading adaptation as posterior inference with shrinkage toward a meta-prior.

We place the conjugate machinery where its Gaussian assumption is most defensible, on features that have already been Gaussianized. We then use the result diagnostically, which none of these papers do.

The closest existing combination is FeCAM-style Tukey-plus-shrinkage and Bayesian quadratic discriminant analysis on frozen features. NIW-Meta reports MetaQDA-MAP, a meta-learned Gaussian classifier with an NIW prior, as a baseline. Our method differs in two ways. The transform itself is task-adaptive and hierarchically shrunk. And the output of interest is not only accuracy but the decomposition of the adaptation gap and a pre-hoc predictor of it.

Method

Let $z \in \mathbb{R}^d$ be a frozen feature vector. For task $t$, apply a per-dimension Yeo-Johnson transform $\psi_{\lambda^{(t)}_j}$ to each coordinate $z_j$, where $\lambda^{(t)}_j$ is the task's exponent for dimension $j$. The exponents are tied across tasks by

$$ \lambda^{(t)}_j \sim \mathcal{N}(\mu_j, \tau_j^2), $$

where $\mu_j$ and $\tau_j$ are meta-learned across training tasks. In a new task, $\lambda^{(t)}$ is set to its posterior mode using the task's pooled support features. Estimation starts from the prior mean and takes one or two Newton steps on the transform likelihood plus the prior term, following the Normality Normalization estimator. With one shot, the estimate stays near $\mu_j$. With sixteen, the data dominate.

On the transformed features $x = \psi_{\lambda^{(t)}}(z)$, each class $c$ is Gaussian with mean $m_c$ and covariance $\Sigma_c$ drawn from a shared meta-learned NIW prior. The posterior predictive for a query is then a multivariate Student-t in closed form. Its heavier tails at low shot are exactly the shrinkage behavior we want.

The meta-parameters are $\mu, \tau$ for the transform and the NIW location, scale, and degrees of freedom. They are fit by maximizing held-out query log-likelihood over training episodes. Adaptation to a new task is then a few Newton steps plus matrix algebra. To keep $d \times d$ covariances tractable, features are first projected to a meta-learned subspace of a few hundred dimensions. We will report sensitivity to this choice.

The diagnostic is the leave-one-out posterior predictive log-likelihood of the support labels, normalized by chance. It is available with no fine-tuning. We define the rigidity residual of a task as

$$ G_t = \mathrm{Acc}_{\text{adapter}}(t) - \mathrm{Acc}_{\text{ours}}(t), $$

where $\mathrm{Acc}_{\text{adapter}}$ is the accuracy of the best low-rank adapter under a matched budget. The shape gap is the difference between our method and a plain linear probe.

Research Program

The work proceeds in three stages. The first stage is purely descriptive: does the optimal transform exponent actually vary across domains and dimensions? If the best per-dimension $\lambda$ on histology, retina and natural-image features all sit near 0.5, then FeCAM's fixed exponent is already right. In that case the hierarchical transform is unnecessary, and the contribution reduces to the gap decomposition and the diagnostic. We will say so and drop the transform prior.

The second stage builds the classifier and measures shape gaps and residual gaps across domains and backbones. The third stage tests whether the frozen-feature diagnostic predicts $G_t$ across tasks, and whether gating fine-tuning on it saves compute without losing accuracy.

Meta-training uses a leave-one-dataset-out protocol so that the prior never sees the test domain. For the medical suite, the prior is learned on four of NCT-CRC, SICAPv2, SkinCancer, MESSIDOR and CheXpert and tested on the fifth. For VTAB-1K, it is learned on two of the natural, specialized and structured groups and tested on the third. The backbones are a natural-image CLIP ViT-B, a DINO ViT and a medical vision-language model of the Quilt-1M type used in the medical few-shot study. The medical backbone matters because the rigidity gap should be smaller for an in-domain backbone, and the diagnostic should track that.

Baselines

The naive reference is a regularized logistic-regression linear probe on raw frozen features, tuned by cross-validation on the support set. It is the probe that Few-shot Adaptation of Medical Vision-Language Models reports at 52.05% on 1-shot histology, where CoOp prompt tuning reaches 46.05%.

The strongest frozen-feature competitor on medical tasks is LP+text from the same paper. It blends visual prototypes with text embeddings and reports 55.6% accuracy on 1-shot histology. We will rerun it on identical splits and seeds. Beating it means exceeding its accuracy at 1 and 4 shots on at least four of five medical datasets without using text. Adding text to our class-mean prior is a natural extension, reported separately.

The most direct statistical competitor is a FeCAM-style classifier. It uses a fixed square-root transform, class covariances with shrinkage, and Mahalanobis distance, tuned on the same meta-training domains we use. FeCAM reports 70.8% on 10-task CIFAR-100 incremental learning. No reported result was found on Lacuna for FeCAM on the medical few-shot suite or VTAB-1K, so we run it ourselves. Distribution Calibration is run the same way. Its reported 68.57% on 5-way 1-shot miniImageNet serves as a sanity check of our reimplementation.

A Bayesian quadratic classifier with a meta-learned NIW prior on untransformed features stands in for NIW-Meta and MetaQDA. It isolates the value of Gaussianizing before the conjugate update. The calibration target on miniImageNet is NIW-Meta's reported expected calibration error of 2.68 with a Conv-4 backbone, versus 8.03 for MetaQDA-MAP.

On the adaptation side, LoRA, ABBA and LiFT serve as capacity references on VTAB-1K. LiFT reports 59.9 average there, versus 55.2 for ProtoNet. All get the same labeled data, and each gets a fixed hyperparameter search of 20 configurations.

Three ablations isolate the key ingredient. The first replaces the hierarchical exponent with a fixed 0.5, with a per-task maximum-likelihood estimate, and with no transform. The second replaces the Yeo-Johnson step with $\ell_2$ normalization onto the unit sphere, as in Mahalanobis++, to test the concern that spherical features conflict with Euclidean Gaussian priors. The third replaces the Student-t predictive with a plug-in point estimate.

Why not just…?

Why not just use FeCAM's fixed square-root transform with covariance shrinkage? This is the real threat. If the optimal exponent barely varies across domains, a fixed 0.5 will match the hierarchical transform. The first-stage measurement and the fixed-exponent ablation settle it directly. If they tie, we report that the shape gap is captured by a fixed transform, and keep FeCAM-plus-NIW as the instrument for the gap decomposition. That decomposition is still new, because FeCAM was never compared against adapter fine-tuning on shifted domains.

Why not just estimate the exponent per task, by maximum likelihood, from the support set or from unlabeled query images? With five support images in 5-way 1-shot, a per-dimension exponent estimate is very noisy. The prior exists to control that noise. Transductive estimation from a 75-image query pool could remove the need for the prior entirely, so we include it as an ablation. If it matches the hierarchical version, we will recommend it as the simpler method in transductive settings, and keep the prior only for the inductive case.

Why not just anchor the linear probe with text, as LP+text does? Text anchoring attacks a different weakness, the lack of a prior over class locations. It does nothing about skew or anisotropy. It also fails when text cannot describe fine visual differences, a limitation the rigidity literature notes for cross-modal anchoring. The comparison against LP+text on identical splits, and the combined variant, show whether the two gains add.

Why not just fine-tune a LoRA adapter, or use LiFT, and skip the diagnosis? On tasks with real capacity deficits, adapters will win, and we expect $G_t > 0$ there. The point is knowing which tasks those are before paying for them. LiFT's gain on Cross-Fit (51.90 versus 46.84 for MAML) comes from sharing across tasks in parameter space, and its paper notes it cannot create features the backbone lacks. Our method offers the same kind of sharing in feature-distribution space at close to linear-probe cost. The gating experiment below tests whether that trade is worth making.

Decisive Experiments

The first check is the gap decomposition. Across the five medical datasets at 1, 4 and 16 shots, we expect the hierarchical transform classifier to recover at least half of the linear-probe-to-adapter gap on the natural-image CLIP backbone. We expect it to recover less on structured VTAB tasks such as counting and depth, where the needed features plausibly do not exist. If the recovered fraction is near zero everywhere, the shape explanation is false, and rigidity in these domains is genuinely about capacity. That is a clean negative result worth reporting.

The second check is predictive. Across the roughly 24 test tasks, pooled over backbones, the Spearman correlation between the frozen-feature diagnostic and $G_t$ should be clearly negative, with a bootstrap interval excluding zero. We also expect $G_t$ to be smaller on the medical backbone than on the natural-image backbone for the medical datasets.

The third check is practical. A policy that fine-tunes an adapter only when the diagnostic falls below a threshold, set on meta-training domains, should reach within one point of always fine-tuning while skipping fine-tuning on most tasks. Wall-clock time and peak memory are reported alongside. The reference points are the 4-second, 800MB profile reported for LP+text, against CoOp's 28GB.

Calibration, measured as expected calibration error on miniImageNet and the medical suite, is reported throughout. The Student-t predictive should be better calibrated than the plug-in and FeCAM variants. A failure there would undercut using its likelihood as a diagnostic.