In Bayesian data analysis, we rarely care about a posterior distribution purely for its own sake. Instead, the posterior is typically an intermediate step toward a downstream goal: computing the expected value of a specific function. Whether calculating the expected return of a financial portfolio, the average failure rate of an engineering system, or a specific "quantity of interest" (QoI) in a physical inverse problem, the mathematical objective is the same:
$$\mathbb{E}_{p(\theta|x)}[f(\theta)] = \int f(\theta) p(\theta|x) d\theta$$
The conventional workflow is strictly two-stage and target-agnostic. First, an inference engine approximates the full posterior distribution $p(\theta|x)$ via Markov Chain Monte Carlo (MCMC) or Variational Inference (VI). Second, samples from this approximation are used to compute the average of $f(\theta)$.
This traditional approach suffers from a fundamental inefficiency: it completely ignores the geometry of $f(\theta)$ during the computationally expensive inference phase. If $f(\theta)$ is highly localized or has its largest values in the tails of the posterior, standard methods will waste immense computational effort mapping out regions of the parameter space that contribute nothing to the final expectation, while missing the regions that matter most.
A emerging paradigm shifts this focus by making inference target-aware. By directly integrating the downstream function into the inference process, this framework bypasses the need to accurately model the entire posterior, unlocking orders-of-magnitude improvements in efficiency and accuracy.
The Pitfall of Target-Agnostic Sampling
To understand why target-aware inference is necessary, consider a scenario where the posterior $p(\theta|x)$ is a standard Gaussian distribution, but the target function $f(\theta)$ is a sharp spike located three standard deviations away from the mean.
Standard Posterior p(θ|x): [---...---( * )---...---] (High probability region)
Target Function f(θ): [---...-----------( ! )---] (High impact region in tail)
A standard target-agnostic sampler will generate the vast majority of its samples near the center of the Gaussian. It may never sample the region where $f(\theta)$ is active, leading to an estimated expectation of zero and catastrophic variance.
To fix this, Adam Golinski et al. introduced Amortized Monte Carlo Integration (AMCI) in Amortized Monte Carlo Integration. The authors demonstrated that by training an amortized proposal distribution to explicitly minimize the variance of the integration process—rather than trying to match the posterior distribution—one can tailor proposals directly to the function being integrated. In fact, if the proposal distribution is shaped proportionally to $|f(\theta)|p(\theta|x)$, the estimator can theoretically achieve zero variance.
This philosophy also applies to high-dimensional deterministic problems. In Goal-Oriented Optimal Approximations of Bayesian Linear Inverse Problems, Alessio Spantini et al. focused on linear inverse problems where the goal is to infer a specific QoI. Rather than constructing a computationally crushing low-rank approximation of the entire posterior covariance matrix, the authors developed optimal low-rank approximations tailored specifically to the subspace that informs the QoI. By ignoring irrelevant dimensions of the parameter space, they drastically reduced computational costs without sacrificing accuracy on the downstream task.
The Core Mechanism: Expectations as Marginal Likelihoods
How do we systematically inject the target function $f(\theta)$ into the inference engine when the posterior itself is unnormalized? A breakthrough insight comes from reframing the expectation problem entirely: an expectation can be decomposed into a ratio of normalization constants (marginal likelihoods).
Recall Bayes' rule for the posterior distribution: $p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$, where $p(x)$ is the marginal likelihood (or evidence). We can substitute this into our expectation formula:
$$\mathbb{E}_{p(\theta|x)}[f(\theta)] = \int f(\theta) \frac{p(x|\theta)p(\theta)}{p(x)} d\theta = \frac{\int f(\theta) p(x|\theta)p(\theta) d\theta}{p(x)}$$
If we assume for a moment that $f(\theta)$ is strictly positive, the term $f(\theta)p(x|\theta)$ can be viewed as a modified likelihood in a pseudo-probabilistic model. The numerator then becomes exactly equivalent to the marginal likelihood of this modified model, which we can denote as $Z_f$. The denominator is simply the marginal likelihood of the original model, $Z_{\text{orig}}$. Thus, the expectation simplifies to a clean ratio:
$$\mathbb{E}_{p(\theta|x)}[f(\theta)] = \frac{Z_f}{Z_{\text{orig}}}$$
This mathematical identity is incredibly powerful. It means that any algorithm capable of estimating the marginal likelihood of a model (such as nested sampling, annealed importance sampling, or thermodynamic integration) can be used to directly compute expectations.
In Target-Aware Bayesian Inference: How to Beat Optimal Conventional Estimators, Tom Rainforth et al. formalized this concept into the Target-Aware Bayesian Inference (TABI) framework. To handle target functions that can take both positive and negative values, the authors decomposed $f(\theta)$ into its positive and negative parts, $f(\theta) = f^+(\theta) - f^-(\theta)$. This splits the target expectation into three distinct component expectations, each of which is estimated using a specialized importance sampling estimator. By breaking the problem apart, TABI bypasses the fundamental variance and error limits of conventional Monte Carlo estimators, achieving arbitrary accuracy with a minimal number of samples.
┌──► Estimate Z_orig (Original Evidence)
│
Target Expectation├──► Estimate Z_f+ (Positive Component Evidence)
│
└──► Estimate Z_f- (Negative Component Evidence)
Automating Target-Aware Inference via Expectation Programming
While mathematically elegant, manually rewriting probabilistic models to incorporate target functions is tedious and error-prone. To bring this capability to everyday scientific workflows, Tim Reichelt et al. introduced Expectation Programming in a pair of foundational papers (AABI 2020; UAI 2022).
Expectation Programming redefines the execution flow of Probabilistic Programming Systems (PPSs). Instead of writing a program that outputs a distribution from which samples are drawn, a user defines an expectation program that directly targets the value whose expectation is sought.
As detailed by the authors in their UAI 2022 paper, the system uses automated source-code transformations to ingest the target function and split the program into multiple marginal likelihood estimation tasks. By treating the positive and negative components of the function as separate evidence components, the compiler automatically constructs a sampling density that prioritizes high-impact regions.
This modular architecture allows practitioners to leverage existing, highly optimized inference backends (like advanced sequential Monte Carlo or variational engines) to compute expectations directly. The system automatically prevents the severe errors that occur when high-impact regions reside in low-probability areas of the standard posterior, providing orders-of-magnitude improvements in both sample efficiency and accuracy.
Summary
Target-aware inference flips the traditional Bayesian paradigm on its head. By recognizing that full posterior estimation is often an unnecessary and inefficient detour, these methods embed the downstream objective directly into the heart of the inference engine. Whether through optimized low-rank approximations, amortized zero-variance proposals, or decomposing expectations into marginal likelihood ratios, target-aware inference ensures that every single compute cycle is spent mapping out the regions of the parameter space that matter most to the final answer.
Key Foundational Work
Amortized Monte Carlo Integration (2019) — contributed 1 concept elements to this direction.
Goal-Oriented Optimal Approximations of Bayesian Linear Inverse Problems (2017) — contributed 1 concept elements to this direction.