Bayesian inference frequently requires evaluating intractable likelihood functions. To bypass this obstacle, modern algorithms rely on approximations. For instance, Approximate Bayesian Computation (ABC) replaces exact data matching with a tolerance threshold, while Sequential Monte Carlo (SMC) and Self-Normalized Importance Sampling (SNIS) approximate continuous target distributions using a finite pool of particles.
However, these approximations introduce systematic statistical bias. In ABC, accepting simulations that are merely "close" to the observed data blurs the posterior distribution. In particle filters, single-pass importance sampling introduces a structural bias that only decays slowly as the particle count increases. To achieve high accuracy, practitioners traditionally had to shrink tolerances to near-zero or scale particle counts to infinity—both of which are computationally prohibitive.
A powerful alternative paradigm has emerged: instead of eliminating the approximation error during sampling, we can accept a larger, computationally efficient approximation error and systematically scrub it away afterward. This is achieved through two complementary mechanisms: post-hoc regression adjustment and iterative Markovian recycling.
Post-Hoc Regression Adjustment: Extrapolating to Zero Error
When using ABC, we generate parameter proposals and simulate summary statistics. We accept a proposal if its simulated statistics fall within a tolerance radius of the observed statistics. Because this tolerance is non-zero, the accepted parameters are biased by the discrepancy between the simulated and observed data.
The core intuition of regression adjustment is to model this discrepancy explicitly. By fitting a local regression model to the accepted parameter-statistic pairs, we can learn exactly how changes in the simulated statistics affect the parameter estimates. Once this relationship is mapped, we can mathematically extrapolate the parameters to the ideal point where the discrepancy is exactly zero.
Accepted Parameters (with error) ───► Local Regression Model ───► Extrapolate to Discrepancy = 0 ───► Unbiased Posterior
As shown by Arnaud Estoup et al. in GENETIC ANALYSIS OF COMPLEX DEMOGRAPHIC SCENARIOS: SPATIALLY EXPANDING POPULATIONS OF THE CANE TOAD, BUFO MARINUS, this rejection-regression framework significantly refines posterior accuracy. Rather than requiring an impossibly small tolerance threshold, Dennis Prangle et al. demonstrate in Diagnostic tools for approximate Bayesian computation using the coverage property that regression adjustments allow practitioners to use much larger, computationally efficient tolerances while maintaining well-calibrated credible intervals. Furthermore, in A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation, Michaël G. B. Blum et al. establish that this post-processing step consistently corrects for the discrepancy between simulated and observed statistics, regardless of the underlying dimension reduction techniques used to compress the data.
This extrapolation principle extends beyond ABC. In distributed Bayesian frameworks like the one proposed by Lewis J. Rendell et al. in Global Consensus Monte Carlo, a regularization parameter $\lambda$ is introduced to manage communication constraints. By running an SMC sampler across a sequence of decreasing $\lambda$ values, the authors use local linear regression to extrapolate the posterior estimates to $\lambda = 0$, effectively neutralizing the bias introduced by the instrumental model. To make these adjustments even more robust, Guilherme S. Rodrigues et al. introduced a "recalibration" step in Recalibration: A post-processing method for approximate Bayesian computation that transforms samples based on marginal distribution functions, ensuring that the final credible intervals possess exact coverage probabilities.
Iterative Smoothing: Turning Inverse Bias into Exponential Decay
While regression adjustment works by mapping and subtracting explicit discrepancies, a second class of methods targets the implicit bias found in particle-based importance sampling. In standard SNIS or single-pass particle filtering, the systematic bias decreases only inversely ($1/N$) with the number of samples or particles.
The breakthrough idea here is to embed these single-pass importance samplers within a Markov Chain Monte Carlo (MCMC) kernel—such as the iterated sampling-importance resampling (i-SIR) or Particle Gibbs algorithms—and recycle the samples across successive iterations.
Why does this work? In a single pass, a particle filter is vulnerable to path degeneracy and weight variance. However, by maintaining a reference trajectory and updating it iteratively through conditional sequential Monte Carlo, the authors of Conditioning diffusion models by explicit forward-backward bridging, Adrien Corenflos et al., ensure asymptotic convergence to the true conditional distribution, eliminating the single-pass bias.
Remarkably, when we average these estimators over the iterations of the Markov chain (after a brief burn-in period), the systematic bias decays exponentially relative to the number of iterations, rather than inversely with the sample size. As proven by Gabriel Cardoso et al. in BR-SNIS: Bias Reduced Self-Normalized Importance Sampling, recycling candidate samples from an i-SIR chain transforms the standard error bounds, achieving exponential bias reduction at virtually zero extra computational cost.
This geometric erasure of bias is highly effective for state-space models. By treating the output of an incremental particle smoother as a step in a Gibbs sampler, Eric Moulines et al. showed in Particle-based, rapid incremental smoother meets particle Gibbs that the Parisian Particle Gibbs (PPG) sampler rapidly scrubs away the linear bias typical of importance sampling. In follow-up work, Gabriel Cardoso et al. extended this to joint state and parameter learning in State and parameter learning with PARIS particle Gibbs, demonstrating that coupling Gibbs iterations with online particle smoothing facilitates exponential bias reduction, ensuring that the systematic error vanishes rapidly as iterations progress.
Recursive Posterior Sharpening
At the extreme end of this iterative paradigm lies the concept of recursive Bayesian updating on the same data. Instead of using iterations merely to scrub sampling bias, one can repeatedly apply Bayes' rule to concentrate the posterior itself.
As explored by Takafumi Kajihara et al. in Kernel Recursive ABC: Point Estimation with Intractable Likelihood, repeatedly feeding the same observed data back into the updating step causes the posterior distribution to collapse into a sharp peak centered exactly at the maximum likelihood estimate. This recursive concentration completely overrides and eliminates the bias that was originally introduced by an incorrect or uninformative prior distribution.
Summary
The shared intuition across these diverse methods is a shift in perspective: we do not need perfect simulations or infinite particles to achieve unbiased Bayesian inference. Whether by fitting a local regression line to extrapolate away a tolerance gap or by leveraging the geometric ergodic properties of Markov chains to exponentially forget importance sampling errors, post-processing and iterative feedback allow us to turn cheap, biased approximations into highly accurate, calibrated posteriors.
Key Foundational Work
A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation (2013) — contributed 2 concept elements to this direction.
GENETIC ANALYSIS OF COMPLEX DEMOGRAPHIC SCENARIOS: SPATIALLY EXPANDING POPULATIONS OF THE CANE TOAD, BUFO MARINUS (2004) — contributed 1 concept elements to this direction.