Label-Free Priors for Certifying Few-Shot Fine-Tuning

When a large pre-trained model is fine-tuned on a few hundred labeled examples, we would like a certificate: a number, valid with high probability, that upper-bounds the fine-tuned model's error on future data. Recent work has produced such certificates only by giving up ordinary fine-tuning. One method picks from a finite menu of pre-made adapters. Another mixes a handful of existing checkpoints. A third quantizes the adapter down to a few dozen bits. This proposal keeps ordinary gradient-based low-rank fine-tuning, meaning small trainable matrices added to a frozen backbone, and certifies it with a PAC-Bayes bound. That is a guarantee whose looseness is set by how far the learned weights move from a "prior" distribution chosen before seeing the labels. The new ingredient is how the prior is built. It uses only information that costs no labels: the pre-trained weights, a classifier head written from the class names (the model's own zero-shot predictor), and the geometry of unlabeled target inputs. Because no labels go into the prior, all labels remain available for the bound.

The proposal also holds itself to a bar that the existing literature largely skips. In the few-shot regime, the simplest certificate is not a PAC-Bayes bound at all. You can take the zero-shot model, which never saw the labels, and bound its error exactly using every labeled example. Or you can hold out a slice of labels and bound the fine-tuned model on it. A fine-tuning certificate is only worth having if it beats both. We propose to build the label-free-prior certificate, measure it against these references on the same vision and language benchmarks used by prior certification work, and map the regime in which each route wins.

Core Claim

Fine-tuning a pre-trained model on few labels can be certified non-vacuously, without discretizing or quantizing the adapter, provided the PAC-Bayes prior already encodes what the model "knows" before labels arrive. Concretely, the prior is a Gaussian over adapter and head weights with three properties. It is centered at the pre-trained backbone plus the zero-shot text-derived head. Its covariance is shaped by the second moments of features on a pool of unlabeled target inputs that is disjoint from the labeled set. And it is independent of the labeled sample, so no labels are spent on it. Our hypothesis is that this certificate is tighter than the best of two references: the zero-shot model certified on all labels, and the fine-tuned model certified on an optimally sized holdout. If it is not, that negative result is itself useful, because it would mean practitioners should simply hold out labels.

The bound we evaluate is the standard small-kl PAC-Bayes inequality. For a posterior $Q$ over weights, a prior $P$ fixed independently of the $n$ labeled examples, and confidence $1-\delta$, it states

$$\mathrm{kl}\big(\hat{L}_S(Q)\,\|\,L(Q)\big) \le \frac{\mathrm{KL}(Q\|P) + \ln(2\sqrt{n}/\delta)}{n},$$

where $\hat{L}_S(Q)$ is the average training error of weights drawn from $Q$, $L(Q)$ is their true error, and $\mathrm{kl}$ is the relative entropy between two Bernoulli distributions. The holdout reference instead inverts a binomial tail on $m$ held-out labels, so its slack shrinks roughly like $\sqrt{1/m}$ with no complexity term. The comparison therefore reduces to a concrete question. Can a label-free prior make $\mathrm{KL}(Q\|P)$ small enough that using all $n$ labels beats spending $m$ of them on a test?

Relation to Prior Work

Several papers supply motivation: they show that certificates for fine-tuning are possible, but only at a cost. Weight-Space Learning for Certifiable Few-shot Transfer Learning (STEEL) replaces gradient descent with sampling a finite set of adapters from a diffusion model trained on an upstream adapter zoo, then applies a union bound. It certifies 99.16% of LaMP-5 tasks versus 0% for SGD, but pays an accuracy gap: 88.40% versus 90.32% on 16-shot CUB. It also depends on a relevant zoo existing. Model Merging is Secretly Certifiable learns only a few merge coefficients over task vectors and reports certified gaps as small as 5% on 100-shot CLIP tasks. It requires relevant checkpoints and splits the scarce labels to build a data-dependent prior. Tight PAC-Bayes Generalisation Guarantees for Large Language Model Safety Monitoring certifies LoRA adapters by compressing them, cutting a Llama-Guard-3-8B adapter's description from 576 to 80 bits. It certifies only that quantized surrogate, and only for binary tasks. Our proposal differs from all three: it keeps continuous gradient fine-tuning, needs no zoo or checkpoint pool, and spends no labels on the prior.

Three papers supply parts of the method. Certifying Deep Network Risks and Individual Predictions with PAC-Bayes Loss via Localized Priors avoids sample splitting by concentrating the prior around low-training-risk weights. It already evaluates GPT-2 LoRA on MRPC, SST-2 and RTE, and certifies ImageNet ResNet-50 at 31.0% against 27.0% true error. It is the closest existing continuous-adapter certificate, and we treat it as a direct competitor. Its prior is built from the labeled training risk through a Gibbs construction. Ours is built from label-free side information and is independent of the labeled sample by construction. Dangers of Bayesian Model Averaging under Covariate Shift shows that isotropic priors put noise into feature directions the data never activates, and proposes shaping the prior by the empirical input covariance. We reuse that idea, but estimate the covariance from unlabeled target inputs so that the certified distribution matches the one we deploy on. On Margins and Derandomisation in PAC-Bayes supplies margin-based tools for transferring a bound on the random predictor $Q$ to its deterministic mean. We use those tools to report a certificate for the model actually deployed.

Other work constrains the design. Improved Regularization and Robustness for Fine-tuning in Neural Networks derives a bound governed by layer-wise distance from pre-trained weights and finds that higher layers move farther. This argues for a per-layer, non-isotropic prior scale rather than a single variance. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning and Sparse is Enough in Fine-tuning Pre-trained Large Language Models argue that fine-tuning lives in a very small effective subspace. That suggests the KL term can be small, but neither computes a certificate. PAC-tuning and Fast Randomized Low-Rank Adaptation with PAC Regularization use PAC-Bayes terms as training regularizers and report accuracy gains, but never report the bound's value. We borrow their bound-driven training objective and add what they omit: the certificate number, compared with a holdout.

Research Program

We would build the prior in three layers and test each separately. The first layer centers the adapter prior at zero, meaning the pre-trained backbone is unchanged. It centers the head at the zero-shot classifier: CLIP text embeddings of class prompts for vision, and verbalizer-token logits for language models. It sets per-layer variances on a fixed grid chosen before seeing labels. Any variance selection that does touch the labels is paid for with a union bound over the grid.

The second layer reshapes the prior covariance of the head and of each adapter's input side. It uses feature second moments computed on an unlabeled pool of target inputs, for example the unused training images of the benchmark with labels discarded. Directions that the unlabeled data never excites get small prior variance, so the posterior cannot hide noise there. Directions with little effect on outputs get large variance, so the posterior can match the prior there at almost no KL cost.

The third layer is the posterior. It is a diagonal or low-rank Gaussian over LoRA and head weights. It is initialized at the result of ordinary fine-tuning on all labels, then refined by directly minimizing the bound above, as in PAC-tuning and the merging paper's direct bound optimization. We report the bound for the random predictor $Q$, and a derandomized bound for its mean via margins.

The analysis side asks where the bits go. We decompose the KL term by layer and by head versus adapter, and we track it as the number of shots grows. From this we would derive the break-even holdout size: the $m$ at which a binomial holdout certificate matches the PAC-Bayes certificate. That quantity tells a practitioner, before running anything, which route to choose.

Baselines

The naive reference point is the zero-shot model certified on all labels. CLIP's text-derived classifier, or the language model with a fixed prompt, never touches the labeled data. Its error can therefore be bounded by a binomial tail over all $n$ examples at $\delta = 0.05$. This is often a surprisingly tight certificate, and the file contains no reported number for it on these benchmarks: no reported result found on Lacuna for zero-shot CLIP certificates on CUB, EuroSAT, GTSRB, DTD or SVHN. We will compute it ourselves. To count as progress, a fine-tuning certificate must be lower than this one, not merely non-vacuous.

The second reference is holdout certification of ordinary fine-tuning. We fine-tune LoRA on $n-m$ labels with the same tuning budget as our method and bound its error on the $m$ held-out labels. We sweep $m$ over a grid and report the best split, so the baseline is at its strongest; the few-shot labeled set is shared across all methods. Beating it means a lower certified error at the same $n$ and $\delta$, together with deployed accuracy no worse than the holdout model.

The strongest existing certified methods are STEEL, model merging, and localized priors. STEEL reports 88.40% accuracy on 16-shot CUB with a non-vacuous certificate; no reported result found on Lacuna for its numeric bound value on CUB. For model merging, the paper reports a certified gap of about 5% at 100 shots on CLIP-ViT-B/32 across EuroSAT, GTSRB, MNIST, SVHN and DTD. We will run merging and STEEL on the same splits, backbone and $\delta$, using the authors' task-vector and adapter pools where they are released. Localized priors will be run on GPT-2 LoRA for MRPC, SST-2 and RTE with matched training compute; no reported per-task bound values for it were found on Lacuna. Beating these means matching or lowering the certified error while keeping the accuracy of unconstrained fine-tuning, and in particular closing STEEL's roughly two-point accuracy gap.

The ablations isolate the key ingredient. First, an isotropic Gaussian prior at the pre-trained weights with a randomly initialized head. Second, the same prior with the zero-shot head added. Third, the zero-shot head plus unlabeled covariance, which is the full method. A fourth arm replaces the unlabeled pool with a label-spending data-dependent prior built from 30–50% of the labels, which directly measures what not spending labels buys.

Why not just…?

Why not just hold out some labels and use a binomial bound? This is the most serious threat, and the proposal is built around it. At 16 shots over 200 CUB classes there are 3,200 labels. Holding out 20% gives a test of 640 examples, whose slack is only a few points, while probably costing little accuracy. It is quite possible that holdout wins in that regime. The experiments settle this with the break-even analysis. We expect PAC-Bayes to win when $n$ is in the tens to low hundreds, as in the 100-sample GLUE and 100-shot vision settings of PAC-tuning and the merging paper, where every held-out label is costly. If it does not win even there, we will report that plainly.

Why not just certify the zero-shot model and skip fine-tuning? Its certificate uses all labels at no complexity cost, so it is the natural floor. The question is whether fine-tuning's accuracy gain outweighs its KL cost. Evidence that fine-tuning moves accuracy substantially is indirect: STEEL's SGD reference reaches 90.32% on CUB, and Return of the Devil in the Details found fine-tuning beats frozen features. Because the proposal's prior is centered at the zero-shot model, KL measures exactly how far fine-tuning departs from it. The comparison is therefore direct, and on tasks where zero-shot is already strong we expect it to win.

Why not just quantize the adapter and use a compression bound, as LoRA-GT does? Compression certifies a modified model rather than the deployed one. Its bit count also ignores what the zero-shot head and unlabeled data already reveal. It remains a strong competitor, however, so we include a LoRA-GT-style arm with a zero-shot head as a shared starting point. If compression plus that head matches our certificate, the covariance-shaping contribution is not needed, and the ablations will show it.

Why not just use localized priors? Localized priors already avoid splitting and already work with LoRA. Our bet is that side information the labels do not contain, namely class names and unlabeled inputs, adds KL savings that a label-derived prior cannot provide. The two ideas are also compatible: a combined arm uses our prior as the base measure of their Gibbs construction.

Proposed Experiments

The vision suite uses CLIP-ViT-B/32 on EuroSAT, GTSRB, DTD and SVHN at 100 shots, matching the merging paper. It also uses CUB, FGVC-Aircraft and Flowers at 16 shots, matching STEEL. The unlabeled pool comes from each dataset's remaining training images with labels removed. The language suite uses GPT-2 and RoBERTa-base with LoRA on MRPC, SST-2 and RTE, at 20, 100 and 200 labeled examples, matching the PAC-tuning and RLoRA settings. It also includes LaMP-5 with Flan-T5 for a direct comparison with STEEL's 99.16% certification rate. All certificates use $\delta = 0.05$. We report both the certified error and the true test error of the deployed deterministic model.

The decisive check is a single comparison per task and shot count. We compare our certificate against the minimum of the zero-shot certificate and the best holdout certificate. The claim is supported if ours is lower on most tasks at 100 labels or fewer, with test accuracy within a point of unconstrained LoRA. The layer-wise KL decomposition should show most of the savings coming from the zero-shot head and the covariance shaping, rather than from the adapter simply staying near zero.

A secondary check targets distribution shift. We build the prior's unlabeled pool from a shifted domain, following STEEL's iNaturalist-style shift and Dangers of Bayesian Model Averaging under Covariate Shift. If the covariance ingredient matters for the reason we claim, certificates should tighten when the pool matches the deployment domain and loosen when it does not.