UNSUPERVISED CONFORMAL INFERENCE: BOOTSTRAPPING AND ALIGNMENT TO CONTROL LLM UNCERTAINTY

Lingyou Pang, Lei Huang, Jianyu Lin, Tianyu Wang, Akira Horiguchi, Alexander Aue, Carey PriebeSubmission to ICLR 2026 · Withdrawn

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Deploying black-box LLMs requires managing uncertainty in the absence of token-level probability or true labels. We propose introducing an unsupervised conformal inference framework for generation, which integrates: generative models, incorporating: (i) an LLM-compatible atypical score derived from response-embedding Gram matrix, (ii) UCP combined with a bootstrapping variant (BB-UCP) that aggregates residuals to refine quantile precision while maintaining distribution-free, finite-sample coverage, and (iii) conformal alignment, which calibrates a single strictness parameter $\tau$ so a user predicate (e.g., factuality lift) holds on unseen batches with probability $\ge 1-\alpha$. Across different benchmark datasets, our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP, while consistently reducing the severity of factuality severity, outperforming lightweight per-response detectors with similar computational demands. The result is a label-free, API-compatible gate for test-time filtering that turns geometric signals into calibrated, goal-aligned decisions.
BibTeX
@misc{pang2025unsupervised,
  author = {Lingyou Pang and Lei Huang and Jianyu Lin and Tianyu Wang and Akira Horiguchi and Alexander Aue and Carey Priebe},
  title = {{UNSUPERVISED CONFORMAL INFERENCE: BOOTSTRAPPING AND ALIGNMENT TO CONTROL LLM UNCERTAINTY}},
  year = {2025},
  eprint = {2509.23002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23002},
}
Figure