TL;DR
This paper introduces Stiefel-Bayes Adapters (SBA), a novel Bayesian approach for parameter-efficient fine-tuning (PEFT) of large language models. SBA leverages a geometric prior on the Stiefel manifold to provide principled uncertainty estimates, leading to more reliable predictions, especially under domain shift, without sacrificing performance or significantly increasing computational cost.
What Problem Does This Paper Address?
Large language models (LLMs) are increasingly adapted for specific tasks using PEFT methods like LoRA. While these methods are efficient, they typically produce point estimates, lacking crucial uncertainty quantification. This deficiency leads to unreliable behavior, particularly when deployed in real-world scenarios where data distributions might shift (domain shift). Existing post-hoc calibration techniques often degrade under such shifts. The paper addresses this by proposing a Bayesian PEFT method that inherently provides calibrated uncertainty estimates by respecting the geometric structure of adapter parameters.
Methods
The core innovation is Stiefel-Bayes Adapters (SBA), which places a Matrix Langevin prior on the Stiefel manifold of orthonormal adapter factors. This geometric prior naturally encodes the inductive bias that adapter subspaces should be well-conditioned and orthogonal. To perform inference, SBA uses a tangent space Laplace approximation with geodesic retraction. This approach avoids the "projection problem" where flat-space Bayesian methods, when projected onto the manifold, suffer from variance inflation and miscalibration. The paper formally proves that this intrinsic manifold inference strictly avoids this issue.
Datasets
The paper evaluates SBA on a range of standard NLP benchmarks, including:
- GLUE and SuperGLUE benchmarks for classification tasks.
- MNLI (Multi-Genre Natural Language Inference) and SST-2 (Stanford Sentiment Treebank) for cross-family and cross-scale generalization.
- Amazon → Yelp and SQUAD → AdversarialQA for domain shift evaluations.
- 20News and IMDB for Out-of-Distribution (OOD) detection.
- XSum for abstractive summarization.
Key Insights and Ideas
- Geometry Matters for Uncertainty: The paper strongly argues that the geometric structure of the adapter parameter space (specifically, the Stiefel manifold for orthonormal factors) is crucial for obtaining reliable uncertainty estimates. Ignoring this geometry leads to miscalibration.
- Intrinsic Manifold Inference is Key: Directly placing priors on the manifold and performing inference within its tangent space, rather than projecting from ambient space, is essential to avoid variance inflation and ensure calibrated posteriors.
- Matrix Langevin Prior is Effective: The Matrix Langevin distribution on the Stiefel manifold provides a principled way to encode prior beliefs about adapter orthonormality and concentration, leading to better posterior distributions.
- SBA Outperforms Ensembles: SBA achieves comparable or better calibration than deep ensembles of LoRA models, but with a fraction of the parameter cost and computational overhead.
Key Results
- Improved Calibration: SBA reduces Expected Calibration Error (ECE) by 18-34% over deterministic baselines and 27-56% over flat-space Bayesian baselines (Gauss+Proj) on various tasks and models.
- Robustness to Domain Shift: SBA significantly improves selective prediction AUROC by 12-25% under domain shift scenarios, demonstrating its ability to reliably identify uncertain predictions.
- Strong OOD Detection: SBA achieves the strongest OOD detection performance, outperforming deep ensembles.
- Competitive Task Performance: SBA matches or slightly trails the best deterministic PEFT methods in terms of task accuracy/F1.
- Computational Efficiency: SBA introduces minimal training overhead (around 7%) and manageable inference overhead, which can be further reduced via predictive distillation.
Limitations and Future Work
- Inference Overhead: SBA introduces inference-time overhead due to posterior sampling. While predictive distillation mitigates this, the distilled model doesn't retain the full uncertainty decomposition.
- Model Scale: The experiments were conducted on models up to 13B parameters; validation on larger models (70B+) is needed.
- Local Approximation: The tangent space Laplace approximation is local and might underestimate uncertainty far from the MAP estimate. Riemannian MCMC methods could address this at higher computational cost.
- Multilingual Settings: The behavior of SBA under multilingual fine-tuning, where adapter subspace structure might differ, is an open question.
- Future Work: The authors suggest applying the tangent space Laplace framework to other Riemannian manifolds (e.g., Grassmannian), extending predictive distillation to preserve full uncertainty, and integrating SBA with active learning.
Summary updated