Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapter

Haotian Xiang, Bingcong Li, Qin LuOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

Abstract

When deploying large language models (LLMs) to safety-critical applications, uncertainty quantification (UQ) is of utmost importance to self-assess the reliability of the LLM-based decisions. However, such decisions typically suffer from overconfidence, particularly after parameter-efficient fine-tuning (PEFT) for downstream domain-specific tasks with limited data. To address these limitations, we build on the Bayesian last layer (BLL) model, where the LLM-based ${\it deterministic}$ feature extractor is followed by random LL parameters for uncertainty reasoning. Since existing low-rank adapters (LoRA) for PEFT have limited expressiveness due to rank collapse, we address this with Polar-decomposed Low-rank Adapter Representation (PoLAR), an orthogonalized parameterization paired with Riemannian optimization to enable more stable and expressive adaptation. The resulting PoLAR-VBLL is a flexible framework that nicely integrates architecture-enhanced optimization with scalable Bayesian inference to endow LLMs with well-calibrated UQ. Our empirical results verify the effectiveness of PoLAR-VBLL in terms of generalization and uncertainty estimation on both in-distribution and out-of-distribution data for various common-sense reasoning tasks.
BibTeX
@misc{xiang2026scalable,
  author = {Haotian Xiang and Bingcong Li and Qin Lu},
  title = {{Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapter}},
  year = {2026},
  url = {https://openreview.net/forum?id=1VCu7aFQzk},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure