TL;DR
This paper empirically investigates whether In-Context Reinforcement Learning (ICRL) models, like the Decision-Pretrained Transformer (DPT), actually behave as Bayesian posterior samplers in practice. By developing statistical tests for "sufficient-statistic invariance" and the "martingale property," the authors demonstrate that these models consistently violate fundamental Bayesian principles, even when their decision-making performance is high.
What Problem Does This Paper Address?
In-Context Reinforcement Learning (ICRL) is a powerful paradigm where a transformer, pretrained on many tasks, can solve a new, unseen task at inference time simply by conditioning on past interactions (the "context"). No parameter updates are needed.
Theoretical work has suggested that under idealized conditions—such as infinite training data and a perfectly expressive model—frameworks like the Decision-Pretrained Transformer (DPT) should act as efficient Bayesian posterior samplers. This means they should implicitly maintain a "belief" about the environment and sample actions based on that belief. However, there is a gap between this theory and practice. This paper asks: Does this Bayesian characterization actually hold up for models trained in the real world?
Methods
The authors establish two necessary conditions that any true Bayesian posterior sampler must satisfy:
- Sufficient-Statistic Invariance: The model's action distribution should only depend on the "sufficient statistics" of the history (e.g., the total reward and number of pulls for each arm in a bandit game), not on the specific order of events or noise that doesn't change those statistics.
- Martingale Property: A model's current action distribution should be the expected value of its future action distributions, assuming it continues to act and learn based on its own internal logic.
To make these properties testable, the authors introduced two new DPT variants:
- DPT': Extends DPT to jointly predict the optimal action and its associated reward.
- GDPT: A generative version that autoregressively predicts the next action-reward pair.
They then developed a statistical testing framework using the paired Hotelling test to determine if practically-trained models significantly deviate from these Bayesian requirements.
Datasets
The researchers evaluated their framework on three classic reinforcement learning benchmarks:
- Gaussian Bandits: Rewards follow a normal distribution with unknown means.
- Bernoulli Bandits: Rewards are binary (success/failure) with unknown probabilities.
- Linear Bandits: Rewards are a linear function of known action features and an unknown shared parameter.
The models were pretrained on large-scale synthetic datasets, ranging from 80,000 to 800,000 environments, to ensure they had ample opportunity to learn the underlying task structures.
Key Insights and Ideas
- Performance is Deceptive: A model can achieve low regret (high cumulative reward) that closely tracks a true Bayesian sampler like Thompson Sampling, yet still fail fundamental Bayesian consistency tests.
- Sensitivity to Context Order: Despite theoretical predictions, transformers are highly sensitive to the order of interactions in the context, violating the invariance principle.
- Internal Inconsistency: The failure of the martingale test suggests that these models do not maintain a self-consistent internal belief state that evolves according to Bayesian logic.
- Architecture Isn't the Only Fix: Even when the authors modified the transformer architecture to be "permutation-invariant" (ignoring order), the models still failed other Bayesian tests, suggesting the issue lies deeper in the training objective or data distribution.
Key Results
- Universal Rejection of Invariance: Across all bandit settings and deployment modes, DPT and its variants consistently failed the sufficient-statistic invariance tests. As shown in the table below, p-values were effectively zero (often $< 10^{-300}$), indicating massive statistical violations.
- Martingale Violations: DPT' and GDPT also failed the martingale property tests in almost every evaluated scenario, further proving their non-Bayesian nature.
- The "Bayesian Gap": While ICRL models can perform well, there remains a persistent gap in the actual action distributions compared to true Bayesian samplers. Figure 1 shows that while DPT's cumulative regret is close to Thompson Sampling (PS), the Total Variation (TV) distance between their action distributions does not vanish.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The study is limited to bandit problems, which lack the state-dependency found in more complex Markov Decision Processes (MDPs).
- The "Why": While the paper proves these models aren't Bayesian, it doesn't fully characterize what non-Bayesian strategies they are using to achieve high performance.
- Future Directions: The authors suggest extending these tests to full MDPs and investigating what specific architectural or training changes are needed to actually induce Bayesian behavior in ICRL models.
This work provides a principled toolkit for researchers to audit "Bayesian" claims in the rapidly evolving field of in-context learning.
Summary updated