Eliciting Bayesian Priors from Observable Predictions

80 concepts64 papers1 proposal

Bayesian inference promises a mathematically rigorous way to fuse human expertise with empirical data. In an ideal workflow, an expert encodes their domain knowledge into a prior distribution $p(\theta)$, data arrives, and a well-calibrated posterior is computed.

In practice, this workflow hits a massive bottleneck. As noted by Mikkola et al. in Prior Knowledge Elicitation: The Past, Present, and Future, prior knowledge elicitation remains the "missing link" of Bayesian statistics. The core issue is a fundamental cognitive disconnect: statistical models are parameterized by abstract mathematical constructs, whereas human experts reason using concrete prototypes, qualitative trends, and observable outcomes. This disconnect is known as the parameter-observable gap.

The Anatomy of the Parameter-Observable Gap

When forced to define a prior in the parameter space, experts frequently struggle. Pfister et al. argue in Exploring the Gap between Informal Mental and Formal Statistical Models that standard statistical interfaces force users to provide mathematical inputs that do not align with how they naturally conceptualize their expectations.

This gap manifests in several ways across different modeling paradigms:

  • Lack of Physical Meaning: In flexible non-Gaussian processes or non-parametric density estimation, parameters often lack any direct physical or statistical interpretation. As Griffin points out in On the Bayesian analysis of species sampling mixture models for density estimation, traditional prior specifications are often chosen for mathematical convenience (such as conjugacy) rather than their relationship to the physical characteristics of the data. This makes it nearly impossible for a researcher to predict how a hyperparameter value will influence the "wiggliness" or multimodality of a density estimate.
  • Qualitative vs. Quantitative Mismatch: Domain experts often possess deep intuitive knowledge but lack the statistical expertise to define formal prior distributions or numerical weights. For instance, Mao et al. observe in Domain Knowledge Uncertainty and Probabilistic Parameter Constraints that humans express expertise as qualitative confidence (e.g., "I'm fairly certain feature A matters more than feature B"), which does not easily map to rigid probability density functions.
  • High-Dimensional Complexity: In complex domains like tabular foundation models or reinforcement learning, defining precise mathematical priors becomes technically infeasible due to the sheer scale of the parameter space. van Leeuwen et al. note in When Data Is Scarce: The Strength of the Prior in Tabular Foundation Models that this difficulty often prevents the effective use of expert knowledge to guide machine learning models in data-scarce environments altogether.

Inverting the Model via the Prior Predictive Distribution

To bridge this gap, a growing body of research shifts the elicitation process entirely from the parameter space to the observable space. Instead of asking an expert to guess an abstract parameter distribution $p(\theta)$, these methods ask the expert to describe the data they expect to see, and then use the model itself to back out the corresponding parameter priors.

The mathematical engine behind this translation is the prior predictive distribution:

$$p(y^*) = \int p(y^* \mid \theta) p(\theta) d\theta$$

Here, $y^*$ represents simulated or observable data. By treating the generative model as a forward mapping from parameters to data, we can frame prior specification as an inverse problem.

Traditional Elicitation:
Expert Intuition ───[Manual Translation]───> Parameter Space p(θ) ───> Model

Predictive Elicitation:
Expert Intuition ───> Observable Space p(y*) ───[Algorithmic Inversion]───> Parameter Space p(θ)

In Flexible Prior Elicitation via the Prior Predictive Distribution, Hartmann et al. demonstrate that querying experts exclusively about observable outcomes prevents them from specifying joint priors that are statistically unreasonable or inconsistent with known data. The model's prior predictive distribution acts as a validator, ensuring that human intuition maps to a mathematically sound parameter space.

Similarly, de Souza da Silva et al. introduce "Prior Predictive Matching" in Prior specification via prior predictive matching: Poisson matrix factorization and beyond. Instead of relying on slow, iterative human feedback, their framework automatically selects hyperparameters by matching the model's theoretical predictions to macro-level data statistics before any training happens, providing an automated solution to the parameter-meaning bottleneck.

Algorithmic Engines of Elicitation

Translating an expert's intuitive predictions into a well-calibrated joint prior requires sophisticated computational machinery to invert the prior predictive integral. Two primary strategies have emerged:

1. Simulation-Based Optimization

In Simulation-based prior knowledge elicitation for parametric Bayesian models, Bockting et al. introduce a model-agnostic framework that treats prior elicitation as an optimization problem. The framework simulates data from the prior predictive distribution and uses stochastic gradient descent (SGD) to automatically "learn" the best hyperparameters for any parametric prior by minimizing the discrepancy between the simulated data patterns and the expert-provided quantiles or moments.

2. Deep Generative Representations

Traditional parametric distributions (like Gaussians or Gammas) are often too rigid to capture the nuanced, highly correlated beliefs of human experts. To overcome this limitation, Bockting et al. leverage deep generative modeling in Expert-elicitation method for non-parametric joint priors using normalizing flows. By querying experts on observable outcomes, they train a Normalizing Flow to warp a simple base distribution into a complex, non-parametric joint prior. This allows the Bayesian model to capture "weirdly shaped" and highly correlated prior beliefs that traditional parametric formulas cannot represent.

Alternative Paradigms for Bypassing the Bottleneck

When full prior elicitation remains out of reach, several adjacent methodologies offer alternative ways to inject domain knowledge without forcing experts to define exact point probabilities.

Posterior Regularization

Rather than forcing domain knowledge into the prior distribution, some frameworks inject expertise directly into the inference step. Zhu 0001 et al. propose "RegBayes" in Bayesian inference with posterior regularization and applications to infinite latent SVMs, a framework that allows for direct regularization of posterior distributions. This bypasses the prior specification bottleneck entirely by enforcing macro-level constraints (such as large-margin classification boundaries) directly on the final posterior.

Active Covariance Learning

In scenarios with small sample sizes where a good prior is critical, systems can actively assist the expert. Afrabandpey et al. design a human-in-the-loop framework in Human-in-the-loop Active Covariance Learning for Improving Prediction in Small Data Sets that actively queries a domain expert about the pairwise similarity of features. The algorithm then constructs an informative prior covariance matrix from these simple comparative judgments, drastically reducing the cognitive load on the expert.

Credal Sets and Robust Bayes

When experts cannot agree on a single prior, or when the data is too sparse to pin down exact numbers, forcing a precise prior can lead to fragile conclusions. Cozman et al. address this in Robustness Analysis of Bayesian Networks with Local Convex Sets of Distributions by replacing exact point probabilities with convex sets of distributions, known as credal sets. This allows for a more relaxed form of elicitation where experts provide bounds rather than exact numbers, yielding robust upper and lower posterior probabilities.

Key Foundational Work

Using Decision Theory to Justify Heuristics (1986) — contributed 1 concept elements to this direction.
Sensitivity Analysis for Probability Assessments in Bayesian Networks (1993) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prior_specification_challenge_1e3c24cc,
  title = {prior specification challenge},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prior-specification-challenge/rdr_1e3c24cc849f48efa77c4d5921df4f16},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_prior_specification_challenge_1e3c24cc_v1,
  title = {prior specification challenge},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prior-specification-challenge/rdr_1e3c24cc849f48efa77c4d5921df4f16/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake