Credal Sets That Become Convex in Logit Space

Uncertainty-aware classifiers try to report two different things. The first is aleatoric uncertainty, meaning the input is ambiguous, as when humans themselves disagree on the label. The second is epistemic uncertainty, meaning the model has not seen enough similar data to know. One principled way to report both is a credal set: instead of one predicted probability vector, the model outputs a set of probability vectors, each consistent with the training data. The spread inside the set reads as epistemic uncertainty, and the ambiguity of its members reads as aleatoric uncertainty. In practice, credal sets for more than three classes are nearly unusable. They are either built by gridding the probability simplex, which only works for three classes, or restricted to axis-aligned boxes, because the natural likelihood-based set is non-convex in probability coordinates. Their size is also hard to measure, because volume collapses in high dimension.

This note proposes moving the problem into the dual coordinate system. Softmax probabilities and logits are related by a Legendre transform, the smooth generalization of the classical point–line duality that maps $(a,b)$ to the line $y=ax-b$. In logit coordinates, the set of last-layer classifiers that fit the training data almost as well as the best one is convex. Its image at any test input is a convex body that we can describe exactly by its support function, the dual description of a convex set that records how far the set reaches in each direction. That one change gives three things at once: input-dependent credal sets for any number of classes, an epistemic measure that does not collapse in high dimension, and conformal coverage guarantees checked by a closed-form membership test rather than a grid.

Core Claim

For a frozen feature extractor $\phi(x)$ with a softmax head $W$, define the plausible set of head perturbations $\mathcal{D}_\alpha=\{\Delta : \ell(W)-\ell(W+\Delta)\le \log(1/\alpha)\}$. Here $\ell$ is the training log-likelihood, $W$ is its maximizer, and $\alpha\in(0,1)$ is a relative-likelihood budget. Because $\ell$ is concave in the head weights, $\mathcal{D}_\alpha$ is convex. The logits at input $x$, $\eta(x)=(W+\Delta)\phi(x)$, are linear in $\Delta$, so the credal set expressed in logit coordinates, $E_\alpha(x)$, is also convex. The same set expressed in probabilities is generally not convex. Under a second-order approximation, $E_\alpha(x)$ is an ellipsoid and its support function has a closed form:

$$h_x(u)=u^\top \hat\eta(x)+\sqrt{2\log(1/\alpha)\,u^\top \Sigma_x u},\qquad \Sigma_x=A_x H^{-1}A_x^\top.$$

In this formula, $u$ is a direction in logit space, $\hat\eta(x)=W\phi(x)$ are the fitted logits, $H$ is the negative Hessian of $\ell$ with respect to the head, and $A_x$ is the linear map from head weights to logits at $x$. Take $u=e_j-e_k$ to get pairwise log-odds directions. Then $h_x(e_j-e_k)$ and $-h_x(e_k-e_j)$ are the exact upper and lower log-odds of class $j$ against class $k$ over the credal set. Each is a single quadratic form, for any number of classes $K$.

We define epistemic uncertainty as the mean pairwise log-odds width, $\mathrm{EU}(x)=\binom{K}{2}^{-1}\sum_{j<k}\big[h_x(e_j-e_k)+h_x(e_k-e_j)\big]$. This is a mean width over a finite direction set. Aleatoric uncertainty is the entropy of the fitted prediction $\mathrm{softmax}(\hat\eta(x))$. The claim is that this pair disentangles better than ensemble mutual information or box credal sets, at single-model cost. Disentangling better means that aleatoric uncertainty tracks human label disagreement while epistemic uncertainty tracks data scarcity and distribution shift.

Why Duality Is the Right Lens

Three findings in the literature motivate the design. Efficient Credal Prediction through Decalibration builds post-hoc credal sets for frozen models by adding a bias vector to the logits, subject to the same relative-likelihood budget. Coupled shifts give non-convex lower probabilities, so it perturbs one logit at a time and outputs probability boxes. We keep its likelihood budget but change coordinates. In logit space, coupling is harmless.

We also note a limitation of its perturbation family that motivates perturbing the head rather than a bias. A single global bias produces the same logit-space set at every input. Any input dependence of its credal sets therefore comes only from softmax saturation, not from how much data supports $x$. Perturbing $W$ makes $\Sigma_x$ grow along feature directions that the training data barely covers.

Conformalized Credal Set Predictors gives distribution-free coverage for credal sets but discretizes the simplex, which confines it to three-class problems such as ChaosNLI. Its authors name closed-form geometric representations as the open problem. In our construction, checking whether a human soft-label distribution $p^*$ lies in the set is a convex membership test: is $\log p^*$, modulo the constant direction softmax ignores, inside $E_\alpha(x)$? Under the ellipsoid it reduces to one Mahalanobis distance, and that distance becomes the conformal nonconformity score.

Is the volume of a credal set a good measure for epistemic uncertainty? proves that volume violates continuity for more than two classes. Volume concentrates at the boundary and vanishes for flat sets. Mean width is a support-function quantity, finite and continuous even for lower-dimensional bodies, which is the dual-envelope property the research file flags in Gap 3. We use it for that reason.

The computational-geometry papers supply the method rather than the application. Approximating smallest enclosing disks shows how a covering question becomes an intersection or membership question under duality. Cache-Oblivious Output-Sensitive Two-Dimensional Convex Hull uses the equivalence between a primal hull and a dual lower envelope. We apply the same move to convert "find the extreme probabilities in a non-convex set" into "evaluate the support function of a convex set."

We deliberately do not build on dual cell enumeration as in An efficient, provably optimal algorithm for the 0-1 loss linear classification problem. Its $O(N^{D+1})$ cost rules out the feature dimensions where uncertainty matters. That constraint pushed us toward a smooth likelihood and an ellipsoidal dual body.

Research Program

We will first establish the exact geometry. We will prove convexity of $E_\alpha(x)$ in logit space and show which credal-set queries become convex programs in these coordinates. Upper and lower pairwise log-odds are convex programs. Upper class probabilities are concave maximizations. Lower class probabilities remain hard, but they admit a valid outer bound from pairwise log-odds maxima. We will also characterize how much the second-order ellipsoid misstates the exact relative-likelihood set.

Second, we will build the scalable estimator. For heads with tens of thousands of parameters, such as CIFAR-100 on 512-dimensional features, $H^{-1}$ is stored in Kronecker-factored form so that $\Sigma_x$ costs one small matrix product per input. On small heads we will also solve the exact non-quadratic program as a reference.

Third, we will add conformal calibration. On held-out examples with human soft labels, the nonconformity score is the Mahalanobis distance of smoothed $\log p^*$ from $\hat\eta(x)$ under $\Sigma_x$. The finite-annotation correction from Conformalized Credal Set Predictors handles noisy empirical label distributions. Zero counts need smoothing, and the sensitivity of the guarantee to that smoothing is part of the analysis.

Finally, we will examine the decomposition under the critique in Rethinking Aleatoric and Epistemic Uncertainty. That paper argues that internal aleatoric estimates often diverge from true data dispersion and that reducible uncertainty should be judged by expected loss reduction. We adopt its standard. Our epistemic measure must predict how much loss drops when more data is added, not merely correlate with distribution shift.

Baselines

The naive reference point is the entropy of the frozen model's softmax, used as a single undifferentiated uncertainty score. Any decomposition has to beat it, on out-of-distribution detection with its epistemic part and on correlation with human disagreement with its aleatoric part.

The strongest sampling-based competitor is a deep ensemble of five independently trained heads on the same frozen features, with epistemic uncertainty measured as mutual information (BALD). We also include a last-layer Laplace approximation that samples from the same Hessian we use. This second baseline isolates whether the credal set and exact support function add anything over Monte Carlo use of identical curvature. All methods share the same features, training split, and hyperparameter budget, with the ensemble given equal search over head regularization.

The strongest existing credal method on our exact setting is the box decalibration of Efficient Credal Prediction through Decalibration. It reports Pareto dominance in coverage-efficiency on CIFAR-10H and "only a few milliseconds of post-processing per class." We found no numerical coverage or efficiency values on Lacuna for it on CIFAR-10H or ChaosNLI, so we will rerun it with the same $\alpha$ grid and calibration split. We beat it if we reach equal empirical coverage of human soft labels with smaller sets, measured by mean log-odds width and by the probability-interval width implied by each set.

For epistemic out-of-distribution detection, the reported reference is Quantifying Aleatoric and Epistemic Uncertainty: A Credal Approach. It reports an AUROC of 0.871±0.017 for FashionMNIST in-distribution against MNIST out-of-distribution, using its log-loss epistemic measure. Matching that while needing a single model counts as parity, and exceeding it counts as a win. On ChaosNLI we also compare against Conformalized Credal Set Predictors, the only setting where its grid is feasible; no reported efficiency number found on Lacuna for it.

Three ablations isolate the key ingredients. The first replaces head perturbation with global bias perturbation in logit space, which tests the claim that input-dependent geometry is what makes epistemic uncertainty track scarcity. The second replaces mean width with log-volume of the same ellipsoid, which tests the volume-collapse argument directly. The third replaces the ellipsoid with the exact relative-likelihood set on small heads, which measures the cost of the approximation.

Why not just…?

Why not just use a deep ensemble and mutual information? Ensembles are the real threat on out-of-distribution detection, and we may only tie them there. Their weaknesses lie elsewhere. They give no set with a coverage guarantee, they cost several training runs, and Rethinking Aleatoric and Epistemic Uncertainty shows BALD estimates poorly exactly when data is scarce. The data-subsampling experiment below settles this. If ensemble mutual information predicts loss reduction as well as our width does at small training sizes, the geometric construction is justified only by coverage and cost, and we will say so.

Why not just sample from a last-layer Laplace posterior? This uses the same Hessian, so it is the closest simple alternative. The difference is that we compute exact extremes of log-odds over a calibrated set instead of Monte Carlo averages under a prior. Sampling underestimates extremes, and nothing calibrates the posterior scale to human label distributions. The Laplace-sampling baseline is run with matched curvature specifically so that, if it matches us on every metric, the conclusion is unambiguous. In that case the contribution reduces to the conformal membership score, which still scales to any $K$ where sampling-based coverage checks do not.

Why not just grid the simplex, or use ordinary conformal prediction sets? Gridding is reported feasible only for three classes in Conformalized Credal Set Predictors, and our Food101 and CIFAR-100 experiments operate at 101 and 100 classes. Ordinary conformal prediction covers labels, not distributions. It cannot separate an input humans find ambiguous from an input the model has never seen, and CLEAR reports conditional coverage of standard conformal quantile regression falling to 40–70% under extrapolation.

Why not just keep decalibration's probability boxes? Boxes discard correlations between classes. With a global bias they also cannot express input-dependent epistemic uncertainty in logit space. The bias-versus-head ablation and the head-to-head efficiency comparison test both points.

Proposed Experiments

CIFAR-10H and ChaosNLI provide human soft labels for calibration and for testing the aleatoric component. Aleatoric entropy should correlate with the entropy of human annotations, while epistemic width should not. FashionMNIST versus MNIST and CIFAR-10 versus SVHN test the epistemic component on distribution shift. Food101 and CIFAR-100 test scaling in $K$ and runtime, where we expect milliseconds per input after a one-time Hessian factorization.

The first decisive check is controlled data subsampling. We train heads on 1% to 100% of the training data. Epistemic width should shrink roughly as $n^{-1/2}$, because $H$ grows linearly in $n$ and the width scales with $H^{-1/2}$. Aleatoric entropy should stay flat, and the shrinkage of epistemic width should predict the observed drop in held-out log-loss.

The second decisive check is synthetic label noise injected into CIFAR-10. Aleatoric entropy should rise with the noise rate, while epistemic width rises only mildly, through the flatter likelihood. If epistemic width moves as much as aleatoric entropy under pure label noise, the decomposition fails, and the method is reduced to a scalable coverage tool rather than a disentanglement method.