Implicit Bayesian Inference in Large Language Models

This direction is split into other directions: implicit Bayesian inference. Its last version stays readable and citable here.

Version 1 of 197 papers124 conceptsfrozen 2026-10-04

When a Large Language Model (LLM) encounters a few examples in a prompt and suddenly "learns" to perform a new task, it isn't updating its weights. This phenomenon, known as In-Context Learning (ICL), has long been the "magic" of the Transformer architecture. However, a growing body of research suggests that this isn't "learning" in the traditional sense. Instead, the model is performing implicit Bayesian inference to identify and activate latent concepts it already acquired during pre-training.

The Model as a Bayesian Reasoner

The core intuition behind this research direction is that an LLM treats its input prompt not just as a sequence of tokens, but as evidence for a hidden state. In An Explanation of In-context Learning as Implicit Bayesian Inference, Sang Michael Xie, Percy Liang, and Tengyu Ma argue that pre-training on diverse data forces the model to represent the world as a mixture of many "latent concepts" or tasks.

When you provide a prompt, the model uses those examples to update its internal "belief" about which task is currently active. It moves from a broad prior distribution (all the things it knows how to do) to a specific posterior distribution (the specific task requested). As Yufeng Zhang and Yixin Wang describe in What and How does In-Context Learning Learn?, this process is mathematically equivalent to Bayesian model averaging, where the model predicts the next token by marginalizing over all possible latent tasks that could explain the prompt.

Locating Tasks instead of Learning Them

If ICL is Bayesian inference, then the "learning" happens during pre-training, not during the prompt. Sewon Min and Hannaneh Hajishirzi provided a famous empirical anchor for this in Rethinking the Role of Demonstrations. They found that replacing correct labels with random ones in a prompt barely degrades performance. This suggests the model isn't learning the mapping between inputs and outputs from the prompt; it is using the format and distribution of the prompt to "anchor" to a latent concept it already possesses.

This "anchoring" is supported by the architecture itself. Boyi Liu and Yixin Wang showed in An Analysis of Attention via the Lens of Exchangeability that the forward pass of an attention layer serves as a mechanism for Bayesian inference, where the output approximates the posterior of a latent variable. Essentially, the Transformer is a machine built to refine its internal theme as it processes more tokens.

Evidence from Dynamics and Scaling

The Bayesian perspective explains several "surprising" behaviors of LLMs:

  1. Phase Transitions: In In-Context Learning Dynamics with Random Binary Sequences, Eric J Bigelow and colleagues observed that models don't improve linearly. Instead, they undergo sudden "phase transitions" where they switch from random guessing to perfect pattern following. This mirrors Bayesian model selection, where the model suddenly finds enough evidence to commit to one specific hypothesis in its "latent library."
  2. Scaling Laws: Aryaman Arora demonstrated in Bayesian scaling laws for in-context learning that treating ICL as approximate Bayesian inference allows us to derive scaling laws that are more interpretable than standard power laws, predicting how performance improves as the "evidence" (context length) increases.
  3. Chain-of-Thought (CoT): Jun Wang argues in Why Can Large Language Models Generate Correct Chain-of-Thoughts? that intermediate reasoning steps act as additional evidence. By generating a CoT, the model "triangulates" the hidden reasoning context, filtering out irrelevant latent variables to converge on the correct reasoning path.

The Deep de Finetti Connection

Why does next-token prediction lead to Bayesian inference? The answer may lie in de Finetti’s theorem. Liyi Zhang, Thomas L. Griffiths, and R. Thomas McCoy show in Deep de Finetti that if a sequence of tokens is exchangeable (meaning their order doesn't change their joint probability), then predicting the next token is mathematically equivalent to performing Bayesian inference on the document's underlying theme.

This suggests that LLM embeddings are not just arbitrary vectors; they are predictive sufficient statistics (Liyi Zhang et al., 2024) that capture the latent distributions used to generate the text.

Implications for Alignment and Forgetting

Viewing LLMs as Bayesian reasoners changes how we approach model safety and fine-tuning. Suhas Kotha and Aditi Raghunathan have shown that "catastrophic forgetting" is often just a failure of task inference. Fine-tuning skews the model's prior, making it "forget" how to access pre-trained skills. However, by using "Conjugate Prompting"—shifting the input distribution away from the fine-tuning data—researchers can recover those "forgotten" capabilities, proving they were internally accessible all along.

Similarly, in the context of weak-to-strong generalization, Felipe Maia Polo and Yaacov Ritov argue that a strong model can use weak supervision as a "pointer" to its own superior latent concepts. By treating a human's noisy labels as evidence for a latent "perfect" task, the model can filter out the noise and align with the supervisor's true intent.

Papers in this version

Showing 60 of 97 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_bayesian_inference_in_large_lan_3cd090ff,
  title = {Implicit Bayesian Inference in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-bayesian-inference-in-large-language-models/rdr_3cd090ffadf047c9a4b8b399783fe09e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_implicit_bayesian_inference_in_large_lan_3cd090ff_v1,
  title = {Implicit Bayesian Inference in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-bayesian-inference-in-large-language-models/rdr_3cd090ffadf047c9a4b8b399783fe09e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}