Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Weiqin Chen, Xinlong Du, Gesualdo Scutari, Harsha HonnappaarXiv 2026 · arXiv

TL;DR

This paper empirically investigates whether In-Context Reinforcement Learning (ICRL) models, like the Decision-Pretrained Transformer (DPT), actually behave as Bayesian posterior samplers in practice. By developing statistical tests for "sufficient-statistic invariance" and the "martingale property," the authors demonstrate that these models consistently violate fundamental Bayesian principles, even when their decision-making performance is high.

What Problem Does This Paper Address?

In-Context Reinforcement Learning (ICRL) is a powerful paradigm where a transformer, pretrained on many tasks, can solve a new, unseen task at inference time simply by conditioning on past interactions (the "context"). No parameter updates are needed.

Theoretical work has suggested that under idealized conditions—such as infinite training data and a perfectly expressive model—frameworks like the Decision-Pretrained Transformer (DPT) should act as efficient Bayesian posterior samplers. This means they should implicitly maintain a "belief" about the environment and sample actions based on that belief. However, there is a gap between this theory and practice. This paper asks: Does this Bayesian characterization actually hold up for models trained in the real world?

Methods

The authors establish two necessary conditions that any true Bayesian posterior sampler must satisfy:

  1. Sufficient-Statistic Invariance: The model's action distribution should only depend on the "sufficient statistics" of the history (e.g., the total reward and number of pulls for each arm in a bandit game), not on the specific order of events or noise that doesn't change those statistics.
  2. Martingale Property: A model's current action distribution should be the expected value of its future action distributions, assuming it continues to act and learn based on its own internal logic.

To make these properties testable, the authors introduced two new DPT variants:

  • DPT': Extends DPT to jointly predict the optimal action and its associated reward.
  • GDPT: A generative version that autoregressively predicts the next action-reward pair.

They then developed a statistical testing framework using the paired Hotelling test to determine if practically-trained models significantly deviate from these Bayesian requirements.

Datasets

The researchers evaluated their framework on three classic reinforcement learning benchmarks:

  • Gaussian Bandits: Rewards follow a normal distribution with unknown means.
  • Bernoulli Bandits: Rewards are binary (success/failure) with unknown probabilities.
  • Linear Bandits: Rewards are a linear function of known action features and an unknown shared parameter.

The models were pretrained on large-scale synthetic datasets, ranging from 80,000 to 800,000 environments, to ensure they had ample opportunity to learn the underlying task structures.

Key Insights and Ideas

  • Performance is Deceptive: A model can achieve low regret (high cumulative reward) that closely tracks a true Bayesian sampler like Thompson Sampling, yet still fail fundamental Bayesian consistency tests.
  • Sensitivity to Context Order: Despite theoretical predictions, transformers are highly sensitive to the order of interactions in the context, violating the invariance principle.
  • Internal Inconsistency: The failure of the martingale test suggests that these models do not maintain a self-consistent internal belief state that evolves according to Bayesian logic.
  • Architecture Isn't the Only Fix: Even when the authors modified the transformer architecture to be "permutation-invariant" (ignoring order), the models still failed other Bayesian tests, suggesting the issue lies deeper in the training objective or data distribution.

Key Results

  • Universal Rejection of Invariance: Across all bandit settings and deployment modes, DPT and its variants consistently failed the sufficient-statistic invariance tests. As shown in the table below, p-values were effectively zero (often $< 10^{-300}$), indicating massive statistical violations.
Table 1: p-values for sufficient-statistic invariance tests on DPT. All values are extremely low, rejecting the Bayesian hypothesis.
  • Martingale Violations: DPT' and GDPT also failed the martingale property tests in almost every evaluated scenario, further proving their non-Bayesian nature.
  • The "Bayesian Gap": While ICRL models can perform well, there remains a persistent gap in the actual action distributions compared to true Bayesian samplers. Figure 1 shows that while DPT's cumulative regret is close to Thompson Sampling (PS), the Total Variation (TV) distance between their action distributions does not vanish.
Figure 1: DPT versus Thompson Sampling in Gaussian bandits: cumulative regret and total vari- ation (TV) distance between action distributions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study is limited to bandit problems, which lack the state-dependency found in more complex Markov Decision Processes (MDPs).
  • The "Why": While the paper proves these models aren't Bayesian, it doesn't fully characterize what non-Bayesian strategies they are using to achieve high performance.
  • Future Directions: The authors suggest extending these tests to full MDPs and investigating what specific architectural or training changes are needed to actually induce Bayesian behavior in ICRL models.

This work provides a principled toolkit for researchers to audit "Bayesian" claims in the rapidly evolving field of in-context learning.

Summary updated

Abstract

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.
BibTeX
@misc{chen2026is,
  author = {Weiqin Chen and Xinlong Du and Gesualdo Scutari and Harsha Honnappa},
  title = {{Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study}},
  year = {2026},
  eprint = {2610.28977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.28977},
}
Figure