Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networks

Joery A. de Vries, Jinke He, Mathijs de Weerdt, Matthijs T. J. Spaan

TL;DR

This paper introduces a novel method, Laplace Variational Recurrent Networks (Laplace VRNN), to equip standard recurrent neural network (RNN) based meta-reinforcement learning agents with Bayesian uncertainty estimation capabilities. By applying the Laplace approximation, they enable these agents to better understand their uncertainty about different tasks without altering their core architecture or requiring retraining, leading to more robust and informative decision-making.

What Problem Does This Paper Address?

Meta-reinforcement learning (meta-RL) aims to train agents that can quickly adapt to new, unseen tasks. A common approach in meta-RL is to use recurrent neural networks (RNNs) to maintain a "point estimate" of the agent's belief about the current task. While effective, this point estimate approach limits the agent's ability to quantify its uncertainty about the task. This lack of uncertainty estimation can lead to overconfidence, brittle performance, and an inability to detect distribution shifts, which are crucial for robust real-world applications. The authors address this by proposing a way to obtain full Bayesian posterior distributions over tasks from these point-estimate based RNNs.

Methods

The core contribution is the Laplace Variational Recurrent Network (Laplace VRNN). This method leverages the Laplace approximation to transform the point-estimate output of an RNN into a full Gaussian distribution representing the task posterior. The key innovation is that this approximation can be applied post-hoc (after training) or during training without modifying the base RNN architecture or its loss function. This is achieved by approximating the curvature of the posterior's log-likelihood at its local maximum, which can be efficiently computed from the RNN's hidden states. The authors explore different factorization assumptions for the posterior and demonstrate that their method can provide useful distributional statistics, like entropy, for non-Bayesian agents.

Datasets

The paper evaluates their method on three problem domains:

  • 1D function regression: A supervised learning task to assess predictive accuracy.
  • Stochastic n-armed bandit: A reinforcement learning task to evaluate decision-making under uncertainty.
  • Deterministic n x n gridworld: Another reinforcement learning task to assess navigation and goal-finding capabilities.

These environments were designed with variations to allow for meta-learning.

Key Insights and Ideas

  • Bayesian Uncertainty for Free: The Laplace approximation offers a computationally efficient way to inject Bayesian uncertainty estimation into existing, non-Bayesian RNN-based meta-RL agents without architectural changes or retraining. This is a significant practical advantage.
  • Observing Overconfidence: The Laplace VRNN allows researchers to observe the overconfidence of standard point-estimate based meta-RL agents. By estimating the entropy of these agents' posteriors, the authors show they often produce overly confident (low entropy) but inconsistent estimators.
  • Post-Hoc Bayesianization: The ability to apply the Laplace approximation after training is a powerful feature, allowing agents to benefit from the stability of deterministic pre-training while still gaining Bayesian insights.
  • Model Assumptions Matter: The paper investigates different factorization assumptions for the posterior, highlighting that certain choices (like summing covariances) are more effective for capturing useful posterior statistics.

Key Results

  • The Laplace VRNN method performs on par with variational baselines in terms of returns on reinforcement learning tasks, but with significantly fewer parameters.
  • In supervised learning tasks (zero-shot regression), the Laplace VRNN shows competitive predictive cross-entropy, especially when applied post-hoc.
  • The method successfully provides useful posterior statistics (e.g., differential entropy, KL-divergence) for non-Bayesian agents, revealing their overconfidence and instability.
  • The "Sum Σ" (summing covariances) variant of the Laplace VRNN consistently showed the most promising results in capturing posterior statistics.

Limitations and Future Work

  • Gaussian Assumption: The Laplace approximation inherently assumes a Gaussian posterior, which might not always be accurate.
  • Jacobian Computation: The computation of Jacobians can be a bottleneck, leading to the need for approximations like using a windowed history.
  • Overconfidence Fix: The method observes overconfidence in non-Bayesian agents but does not fix it. Future work could explore how to leverage these insights to improve agent robustness.
  • Representation Learning: The authors suggest future work on enabling statistically sound representation learning and improving exploration through techniques like distribution-shift detection.
  • Generalization: Investigating the induced biases of the Laplace approximation for more complex posterior distributions is also a direction for future research.

Summary updated

Abstract

Meta-reinforcement learning trains a single reinforcement learning agent on a distribution of tasks to quickly generalize to new tasks outside of the training set at test time. From a Bayesian perspective, one can interpret this as performing amortized variational inference on the posterior distribution over training tasks. Among the various meta-reinforcement learning approaches, a common method is to represent this distribution with a point-estimate using a recurrent neural network. We show how one can augment this point estimate to give full distributions through the Laplace approximation, either at the start of, during, or after learning, without modifying the base model architecture. With our approximation, we are able to estimate distribution statistics (e.g., the entropy) of non-Bayesian agents and observe that point-estimate based methods produce overconfident estimators while not satisfying consistency. Furthermore, when comparing our approach to full-distribution based learning of the task posterior, our method performs similarly to variational baselines while having much fewer parameters.
BibTeX
@inproceedings{vries2025bayesian,
  author = {Joery A. de Vries and Jinke He and Mathijs de Weerdt and Matthijs T. J. Spaan},
  title = {{Bayesian Meta-Reinforcement Learning with Laplace Variational Recurrent Networks}},
  booktitle = {RLC},
  year = {2025},
  url = {https://openreview.net/forum?id=YYw6MlEPaU},
}
Figure