Robust Bayesian inference in complex models with possibility theory

Jérémie Houssineau, David J. NottarXiv 2022 · DOI · arXiv

TL;DR

Researchers have developed a new framework for robust Bayesian inference by swapping standard probability theory for possibility theory. By leveraging a unique property where the "marginal likelihood" measures the consistency between data and priors, the authors created an automated way to discount outliers without needing complex tuning parameters or heavy computational overhead.

What Problem Does This Paper Address?

Standard Bayesian inference is notoriously sensitive to outliers. As George Box famously noted, "all models are wrong," but in the real world, data is often contaminated by noise or anomalies that don't fit the model at all.

In high-dimensional or massive datasets, manually removing these outliers is impossible. Existing robust methods usually rely on "likelihood discounting" (tempering the influence of data), but these often require researchers to manually pick a "tuning parameter" to decide how much to discount. This paper seeks a general, automated solution that works in complex settings—like tracking moving objects or detecting shifts in time-series data—without making the math significantly harder or slower.

Methods

The authors reformulate Bayesian inference using Possibility Theory. While probability theory requires everything to sum to one, possibility functions are defined by their maximum value (which is 1).

The key technical innovation is how they handle the "evidence" (the denominator in Bayes' theorem):

  1. Consistency as a Metric: In possibility theory, the marginal likelihood (the "evidence") directly assesses how much the prior and the likelihood agree.
  2. Automated Discounting: The authors use this consistency measure to calculate a "discount" factor ($\gamma$). When a new data point is highly inconsistent with the current model (likely an outlier), $\gamma$ becomes small, automatically "tempering" that data point's influence on the posterior.
  3. Possibilistic Conjugacy: They show that this approach still works with standard conjugate priors (like Gaussian, Beta, and Inverse-Wishart), allowing for closed-form solutions that are computationally efficient.

Datasets

The paper evaluates the method on three distinct scenarios:

  • Simulated Extended Feature Estimation: A 10-dimensional matrix estimation task designed to test how the model handles inflated covariance (noise).
  • Simulated Kalman Filtering: A dynamical system tracking a "nearly-constant velocity" trajectory contaminated with 10% outliers.
  • Well-Log Data: A classic real-world univariate time-series dataset (from the oil and gas industry) that contains both sudden "change points" (real shifts in the underlying process) and random outliers.

Key Insights and Ideas

  • The "Evidence" Flip: In standard Bayes, the marginal likelihood is often just a normalizing constant. Here, it is repurposed as a "conflict detector."
  • MAP Coherence: In possibility theory, the Maximum A Posteriori (MAP) estimate transforms coherently under re-parameterization. In standard probability, if you change the units or scale of your parameters, your MAP might shift; here, it stays logically consistent.
  • Sequential Processing: The authors found that processing observations one-by-one is crucial for robustness. If you update in large batches, a single outlier can "poison" the consistency measure for the entire batch.
  • No Jacobian Required: Unlike probability density functions, possibility functions don't require a Jacobian term during a change of variables, simplifying the math for complex model transformations.

Key Results

  • Robustness to Noise: In the 10D feature estimation task, the robust possibilistic method maintained a low error rate (Frobenius norm) even as outliers were introduced, whereas standard Bayesian inference error spiked significantly.
  • Tracking Performance: In Kalman filtering simulations, the "discount" method closely matched the performance of a theoretical "perfect" filter that knows exactly which points are outliers.
  • Change-Point Accuracy: On the well-log data, the method successfully distinguished between rapid shifts in the mean (change points) and isolated outliers.
  • Computational Efficiency: The robust method is only marginally slower than standard Bayes. For the well-log data, it took ~12ms per observation compared to ~7ms for the non-robust version—a negligible cost for the added reliability.

Limitations and Future Work

  • Ordering Sensitivity: Because the model updates sequentially, the order of data can technically change the result. While simulations showed this effect was weak, it remains a theoretical limitation.
  • The "First Observation" Problem: The current framework generally assumes the very first data point is an "inlier" to establish a baseline. The authors suggest future work is needed to allow the prior itself to be discounted if the very first data points conflict with it.
  • Unbounded Likelihoods: If a likelihood is unbounded (e.g., a variance of zero), the math breaks. The authors proposed a "splitting" method in Section 4.3 to handle this, but it adds some complexity.
  • Tuning for "Soft" Support: For models with hard boundaries (like Uniform distributions), the authors had to introduce a "soft" version with an exponential decay, which re-introduces a single tuning parameter ($c$).

Summary updated

Abstract

We propose a general solution to the problem of robust Bayesian inference in complex settings where outliers may be present. In practice, the automation of robust Bayesian analyses is important in the many applications involving large and complex datasets. The proposed solution relies on a reformulation of Bayesian inference based on possibility theory, and leverages the observation that, in this context, the marginal likelihood of the data assesses the consistency between prior and likelihood rather than model fitness. Our approach does not require additional parameters in its simplest form and has a limited impact on the computational complexity when compared to non-robust solutions. The generality of our solution is demonstrated via applications on simulated and real data including matrix estimation and change-point detection.
BibTeX
@misc{houssineau2022robust,
  author = {Jérémie Houssineau and David J. Nott},
  title = {{Robust Bayesian inference in complex models with possibility theory}},
  year = {2022},
  eprint = {2204.06911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.06911},
}
Figure