Abstract
Bayesian inference is often used to quantify uncertainty. Several recent analyses have rigorously decomposed uncertainty in prediction by Bayesian inference into two types: the inherent randomness in the data generation process and the variability due to lack of data respectively. Existing studies have analyzed these uncertainties from an information-theoretic perspective, assuming the model is well-specified and treating the model parameters as latent variables. However, such information-theoretic uncertainty analysis fails to account for a widely believed property of uncertainty known as sensitivity between test and training data. This means that if the test data is similar to the training data in some sense, the uncertainty will be smaller. In this study, we study such sensitivity using a new decomposition of uncertainty. Our analysis successfully defines such sensitivity using information-theoretic quantities. Furthermore, we extend the existing analysis of Bayesian meta-learning and show the novel sensitivities among tasks for the first time.
Introduction
In Bayesian machine learning, uncertainty is typically categorized into two forms: aleatoric (inherent noise in the data generation process) and epistemic (uncertainty stemming from a lack of data). Recent information-theoretic frameworks have established a link between epistemic uncertainty and Conditional Mutual Information (CMI). While these frameworks provide a solid foundation, they have historically failed to account for "sensitivity"—the intuitive notion that a model should be more certain about a test point if it closely resembles a point in the training set.
Standard linear models and Gaussian Processes naturally exhibit this geometric property. However, existing CMI-based theories treated the training set as a monolithic block of information, overlooking the specific relationships between individual training points and test data. This paper addresses this gap by formally defining "test data sensitivity" through a novel information-theoretic decomposition.
Methodology: Decomposing Uncertainty
The core contribution of this work is a novel decomposition of CMI (Theorem 1). Rather than treating CMI as an indivisible value, the authors decompose it into three primary components:
- Mutual Information (MI): The information shared between the model parameters and the training set.
- Sensitivity Terms ($I_n$): These terms quantify the specific information a single training point $Z_n$ provides about a test point $Z^*$.
- Interaction Terms: These represent the information shared between different training points.
Unlike previous approaches that relied on upper bounds, this decomposition provides an exact identity. This framework is further extended to Bayesian Meta-learning, introducing the concept of "task sensitivity"—a measure of how much information one training task contributes to the understanding of a new, unseen test task.
Theoretical Insights
By grounding the "geometric" intuition of proximity in information theory, the authors show that sensitivity is formally defined as the mutual information between a test point and a training point, conditioned on the remainder of the dataset.
A significant advantage of this new decomposition is that it yields tighter bounds on uncertainty. By explicitly subtracting positive sensitivity terms from the total mutual information, the resulting upper bound is more precise than those found in prior literature. Furthermore, the authors demonstrate that the relationship between training and test error can be expressed as an exact identity when these sensitivity terms are incorporated, rather than a simple inequality.
Key Results and Experimental Validation
The theoretical findings were validated using synthetic experimental setups, including linear regression with Gaussian and polynomial basis functions, and a hierarchical Bayesian meta-learning model.
As illustrated in Figure 1, numerical experiments confirm that the proposed bound is significantly closer to the actual CMI than previous state-of-the-art bounds. The analysis also reveals critical asymptotic behavior: sensitivity terms ($I_n$) vanish at a rate of $o(1/N)$, which is faster than the standard MI rate of $O(1/N)$. This explains why sensitivity is most impactful in low-data regimes. In meta-learning contexts, "Minimum Excess Meta-Risk" (MEMR) was shown to decrease as the number of tasks ($M$) and data points per task ($N$) increase, with task sensitivity decaying most rapidly.
Conclusion and Future Work
This study provides a rigorous information-theoretic foundation for test data sensitivity in Bayesian inference. While the current analysis assumes well-specified models, it opens several avenues for future research. Key challenges include extending this framework to model misspecification, addressing the computational complexity of calculating CMI in deep neural networks, and generalizing the exact characterization beyond log-loss functions. Future work will also investigate how approximation methods, such as variational inference, influence these sensitivity measures.
Summary updated