Automating Expert Prior Elicitation for Bayesian Models

This direction is split into other directions: prior specification challenge. Its last version stays readable and citable here.

Version 1 of 180 papers87 conceptsfrozen 2026-10-04

Bayesian inference is a powerful framework for decision-making under uncertainty, but it suffers from a persistent "knowledge acquisition bottleneck." To function, Bayesian models require a prior distribution—a mathematical representation of what we believe before seeing any data.

While domain experts possess deep intuition, they rarely think in terms of abstract statistical parameters like "Gamma distribution shape factors" or "covariance matrices." This disconnect creates a barrier where valuable human expertise is left on the table, replaced by uninformative "default" priors that force models to learn everything from scratch.

Modern research is bridging this gap by moving away from asking experts for numbers and instead building systems that can "translate" human intuition into formal math.

The Elicitation Bottleneck

The fundamental problem, as identified as early as 1986 by Curtis Langlotz and Edward H. Shortliffe in Using Decision Theory to Justify Heuristics, is that forcing experts to provide precise numerical probabilities introduces artificial precision. Humans are better at providing "reasons" or "endorsements" for their beliefs rather than exact degrees of certainty, a concept explored by Michael Sullivan and Paul R. Cohen in An Endorsement-based Plan Recognition Program.

In modern contexts, this bottleneck is even more acute. As Jun Zhu notes in Big Learning with Bayesian Methods, scaling Bayesian methods to massive datasets is hindered by the manual effort required to formalize expert knowledge.

From Parameters to Observables

A key breakthrough in this field is Prior Predictive Elicitation. Instead of asking an expert, "What is the mean of the latent variable $\theta$?", researchers ask, "What do you expect the actual data to look like?"

Arto Klami argues in Flexible Prior Elicitation via the Prior Predictive Distribution that because model parameters are often abstract mathematical constructs, experts cannot grasp them intuitively. However, they can easily predict observable outcomes. By matching these predicted outcomes to the model's internal math—a process called Prior Predictive Matching—the system can automatically back-calculate the necessary parameter priors Eliezer de Souza da Silva.

This shift aligns with how humans naturally reason. As Johanna Beyer explains in Exploring the Gap between Informal Mental and Formal Statistical Models, humans reason using concrete prototypes rather than abstract distributions. Interfaces that elicit these "mental models" are far more effective than those requiring statistical inputs.

Large Language Models as Expert Interfaces

The most recent shift involves using Large Language Models (LLMs) to bypass the human interview process entirely. Frameworks like AutoElicit, developed by Alexander Capstick and Rahul G Krishnan, treat LLMs as repositories of domain expertise.

By prompting an LLM to provide "expert-like" priors for clinical trials or predictive models, researchers can achieve high accuracy with significantly less labeled data Using Large Language Models for Expert Prior Elicitation in Predictive Modelling. Shota Arai suggests that this approach could reduce the required sample size in clinical trials by up to 20%, potentially saving lives by accelerating drug discovery How many patients could we save with LLM priors?.

Robustness to Incorrect Expertise

A major fear in using expert priors is that the expert might be wrong. If a prior is too "strong," it can blind the model to what the data is actually saying.

To solve this, researchers like A. Cisse and Andrew I. Cooper developed HypBO (Hypothesis-based Bayesian Optimization). This framework treats expert "hunches" as soft constraints. If the expert is right, the model finds the optimum 10x faster; if the expert is wrong, the model's bilevel optimization structure allows it to "recover" and find the true global optimum regardless HypBO: Accelerating Black-Box Scientific Experiments Using Experts' Hypotheses.

Similarly, Yi Mao and Guy Lebanon proposed probabilistic parameter constraints that allow models to benefit from expert insight when it's accurate but gracefully ignore it when the data provides contradictory evidence Domain Knowledge Uncertainty and Probabilistic Parameter Constraints.

Capturing Complex Beliefs

Traditional priors are often limited to simple shapes like Gaussians. However, human beliefs are often correlated, skewed, or "weirdly shaped."

Florence Bockting and Paul‐Christian Bürkner have pioneered the use of Normalizing Flows to represent these complex joint priors. By using deep learning to warp simple distributions into complex ones based on expert feedback, they can capture nuanced, non-parametric beliefs that were previously impossible to formalize Expert-elicitation method for non-parametric joint priors using normalizing flows.

By combining these three pillars—predictive elicitation, LLM-based automation, and robust optimization—the field is transforming prior elicitation from a "forgotten" theoretical step into a practical, scalable tool for modern AI.

Papers in this version

Showing 60 of 80 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automating_expert_prior_elicitation_for__5200f436,
  title = {Automating Expert Prior Elicitation for Bayesian Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automating-expert-prior-elicitation-for-bayesian-models/rdr_5200f43673ad4188a0f1a15b87af8866},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_automating_expert_prior_elicitation_for__5200f436_v1,
  title = {Automating Expert Prior Elicitation for Bayesian Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automating-expert-prior-elicitation-for-bayesian-models/rdr_5200f43673ad4188a0f1a15b87af8866/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}