Label Descriptions as Bayesian Priors for Calibrated Few-Shot Classification

Few-shot classifiers that read the class name ("born in", "Sports", "Cardiology") get a large accuracy boost from that text. But they use it as a fixed target vector and score inputs by cosine similarity or an entailment probability. That makes them overconfident and unable to say "this matches none of the classes." Bayesian few-shot methods have the opposite profile. They produce honest uncertainty that shrinks as examples arrive, but they treat classes as anonymous integers and so have nothing to say before the first example is seen.

We propose a simple bridge. Treat the embedding of each class's natural-language description as a prior belief about where that class's examples will fall in a frozen text-embedding space. Update that belief in closed form as labeled examples arrive, and read off a predictive distribution whose confidence grows with evidence. The same machinery handles a "none of the above" (NOTA) input: it is an input that is poorly explained by every class, under a class-agnostic background hypothesis. No ad-hoc threshold is needed.

Core Claim

A conjugate Bayesian classifier whose class priors come from label descriptions should do three things. First, it should be calibrated at zero shots. Second, it should avoid the "low-shot hump," the finding by A Study on the Calibration of In-context Learning that the first one to three in-context demonstrations raise an LLM's accuracy while making its calibration worse (ECE about 0.14 at 1-shot on SST-2 for LLaMA-7B). Third, it should detect NOTA inputs without tuning a threshold on a development set.

The specific novelty is narrow and testable. Each class prior is placed and sized by its description, and that placement is meta-learned. No prior work in the file does both halves:

Research Program

We fix a frozen sentence encoder $f$. Each input $x$ is mapped to $z = P f(x)$, where $P$ is a learned projection down to about 64–128 dimensions. This keeps covariance estimation tractable, which is the dimensionality bottleneck noted for Tractable Uncertainty-Aware Meta-Learning.

Each class $c$ has a description $d_c$. We paraphrase it several times with an LLM in a one-time step, as StructCBM does, and embed each paraphrase. The class-conditional distribution is a Gaussian $\mathcal{N}(\mu_c, \Sigma_c)$ with a Normal-Inverse-Wishart prior:

$$ \mu_c \mid \Sigma_c \sim \mathcal{N}\!\left(W \bar e_c + b,\ \Sigma_c/\kappa_c\right), \qquad \Sigma_c \sim \mathcal{IW}(\Psi, \nu). $$

The symbols are as follows:

  • $\bar e_c$ is the mean projected embedding of the paraphrases of $d_c$.
  • $W, b$ are a learned map. It corrects the systematic offset between where descriptions land and where instances land.
  • $\kappa_c$ is the strength of the prior, in units of pseudo-examples.
  • $\Psi$ and $\nu$ are a shared prior scale matrix and degrees of freedom.

We make the prior strength depend on how ambiguous the description is: $\kappa_c = g(s_c)$, where $s_c$ is the spread of the paraphrase embeddings and $g$ is a small monotone network. A vague label like "other organization" therefore gets a weak prior, and a crisp one gets a strong prior.

Given $k$ support examples, the posterior is closed-form. The predictive distribution $p(z \mid c, \text{support})$ is a multivariate Student-t. Its spread narrows as $k$ grows, and that narrowing is what should give confidence that tracks the number of shots. Class probabilities follow from Bayes' rule.

For NOTA we add a background hypothesis $c_0$. Its prior mean is the pooled mean of meta-training classes, and its prior strength is weak. An input is labeled NOTA when $c_0$ has the highest posterior. The decision rule therefore falls out of the model rather than from a dev-set threshold.

The parameters $P, W, b, g, \Psi, \nu$ are fit by maximizing predictive log-likelihood over meta-training episodes. Each episode samples classes, samples a shot count $k \in \{0,\dots,16\}$, and samples queries. We include NOTA queries drawn from held-out meta-training classes. Training takes minutes, because no inner loop or sampling is needed.

Baselines

The naive reference point is the description-only classifier. It takes a softmax over cosine similarities between $f(x)$ and the description embeddings, the dual-encoder matching style the file attributes to Bapna et al. Zero-shot it uses only descriptions; with support examples it adds the support mean to the description embedding. It is run with the same encoder and the same paraphrase sets, with temperature fit on meta-training episodes. If our model does not beat this, the Bayesian layer has added nothing.

The strongest calibration competitor is MetaQDA (Shallow Bayesian Meta Learning for Real-World Few-Shot Recognition). It is the existing method closest to ours, and it reports cutting ECE by factors of 2 to 10 on Meta-Dataset. We run it on the identical projected text features with its shared, description-free prior, and with the same meta-training episodes and tuning budget.

Comparing against MetaQDA isolates the key ingredient, description-conditioned priors. Two wins would count as beating it. The first is lower ECE and NLL at $k \le 4$ on held-out FewRel classes. The second is any non-trivial zero-shot performance, where MetaQDA is undefined.

Two further ablations isolate the remaining design choices. One fixes $\kappa_c$ to a constant to test the ambiguity-scaled prior strength. The other replaces the background class with a dev-tuned threshold.

For accuracy, two baselines matter:

We will reuse their templates and their DeBERTa NLI model rather than reimplementing loosely. We do not expect a frozen small encoder to beat a cross-encoder like DeBERTa-xxLarge on raw F1. The goal is to stay within a few points while giving calibrated probabilities and threshold-free NOTA detection.

For calibration on these relation benchmarks, no reported result was found on Lacuna for ECE on FewRel or TACRED for any method. We will therefore compute ECE for BERT-PAIR and the NLI model ourselves.

The in-context LLM baseline is a frozen LLaMA-family model prompted with $k$ demonstrations. We compare against the reported ECE of about 0.14 at 1-shot on SST-2 and rerun it on our splits, together with the scaling-binning recalibration that paper found effective.

Why not just…?

Why not just apply temperature scaling to the cosine classifier? A single temperature cannot make confidence depend on how many examples back each class. A Study on the Calibration of In-context Learning found temperature scaling insufficient in the low-shot regime. Still, a per-shot-count temperature is a cheap and fair competitor, so we include it in the description-only baseline. The test is whether ECE at matched $k$ differs. Where our method wins, the advantage should show up in the per-input variance on inputs far from the support examples, not in averages.

Why not just shrink the support mean toward the description embedding and skip the Bayesian machinery? This is the real threat. The posterior mean of our model is such a shrinkage estimator. If deterministic shrinkage plus per-$k$ temperature matches our ECE and NLL, the honest conclusion is that only the description-informed mean matters. We would then report that.

The Bayesian version should differ in two specific places. One is NOTA detection, where predictive spread and the background hypothesis matter. The other is classes with ambiguous descriptions, where $\kappa_c$ matters. We build exactly this shrinkage baseline, with the same learned $W, b$ and an optimized shrinkage weight, to settle it.

Why not just prompt an instruction-tuned LLM with the descriptions? The same calibration study reports that instruction tuning and chain-of-thought both raise overconfidence. StructCBM shows that description-based classification can stay cheap at test time, reaching 85.45% on AGNews at 10 shots with 173 LLM calls in total. We still run the LLM baseline. If it is both accurate and calibrated after scaling-binning, our contribution reduces to cost and threshold-free NOTA detection.

Why not just use the NLI entailment score with a tuned threshold? That method needs a labeled development set to set the no-relation threshold. That set is exactly what a true few-shot user lacks, and its probabilities are uncalibrated point estimates. We measure how its NOTA F1 degrades when the threshold is tuned on 0, 16, or full dev data, against our threshold-free rule.

Proposed Experiments

Meta-training uses the 64 FewRel training relations with their Wikidata relation descriptions (FewRel: A Large-Scale Supervised Few-shot Relation Classification Dataset with State-of-the-Art Evaluation). Evaluation covers four settings:

  • the 20 held-out FewRel relations at 5-way and 10-way, for $k \in \{0,1,2,4,8,16\}$;
  • FewRel 2.0 with 0%, 15%, and 50% NOTA, plus its biomedical domain-shift split;
  • TACRED in zero-shot and the 5% few-shot split, using its natural no-relation class;
  • AGNews and SST-2, for direct comparison with StructCBM accuracy and in-context-learning ECE.

The metrics are accuracy or F1, ECE, NLL, and Brier score. For NOTA we also report AUROC and F1 at the model's own decision rule.

Three results are decisive:

  1. ECE across shot counts. Our ECE-versus-$k$ curve should be flat or decreasing from $k=0$, with no hump. The description-only baseline and in-context learning should show the hump on the same splits.
  2. NOTA at 50%. At 50% NOTA on FewRel 2.0, our threshold-free rule should lose clearly fewer than the 7–8 accuracy points BERT-PAIR loses between 0% and 50% NOTA.
  3. Ambiguous classes. On classes binned by paraphrase spread, the gap between our model and the shrinkage baseline should grow with ambiguity.

If the third result fails and the second does not, the paper becomes "description priors plus a background hypothesis." If both fail, the shrinkage baseline wins and we say so.