Abstract
Meta-reinforcement learning trains a single reinforcement learning agent on a distribution of tasks to quickly generalize to new tasks outside of the training set at test time. From a Bayesian perspective, one can interpret this as performing amortized variational inference on the posterior distribution over training tasks. Among the various meta-reinforcement learning approaches, a common method is to represent this distribution with a point-estimate using a recurrent neural network. We show how one can augment this point estimate to give full distributions through the Laplace approximation, either at the start of, during, or after learning, without modifying the base model architecture. With our approximation, we are able to estimate distribution statistics (e.g., the entropy) of non-Bayesian agents and observe that point-estimate based methods produce overconfident estimators while not satisfying consistency. Furthermore, when comparing our approach to full-distribution based learning of the task posterior, our method performs on par with variational baselines while having much fewer parameters.
Introduction
In Meta-Reinforcement Learning (meta-RL), agents are designed to adapt rapidly to novel tasks by leveraging experience from a broad task distribution. Most contemporary meta-RL frameworks utilize Recurrent Neural Networks (RNNs) to process historical states and rewards into a hidden state. This hidden state effectively serves as a point-estimate of the current task. However, point estimates are inherently overconfident and fail to quantify uncertainty, leading to brittleness when the agent encounters distribution shifts or ambiguous task identities.
While Bayesian alternatives like Variational Recurrent Neural Networks (VRNNs) address this by learning explicit distributions, they introduce significant training complexity, sampling noise, and increased parameter counts. This paper introduces the Laplace Variational Recurrent Network (Laplace VRNN), a method to "Bayesian-ify" standard meta-RL agents without architectural modifications.
Methodology
The Laplace VRNN applies the Laplace approximation to the hidden states of a standard RNN. This approach allows for the extraction of a posterior distribution from the curvature of a deterministic model's learned representations.
Curvature Analysis and the Jacobian Shortcut
The core of the method involves a second-order Taylor expansion (the Hessian) of the log-posterior to approximate the task distribution as a Gaussian. To maintain computational efficiency, the authors utilize a "Jacobian shortcut," where the inverse covariance of the Gaussian is computed using the sum of the outer products of the RNN's Jacobians.
Temporal Aggregation
The authors explore two primary variants for aggregating uncertainty over time:
- Stationary: Treats individual time steps independently.
- Markovian (Sum $\Sigma$): Accumulates information over the entire trajectory, mimicking the behavior of a true Bayesian agent by showing decreasing entropy as more data is gathered.
Notably, this approximation can be applied post-hoc, allowing researchers to add uncertainty quantification to pre-trained, non-Bayesian agents.
Experimental Results
The Laplace VRNN was evaluated across three domains: 1D supervised regression, a stochastic 5-armed bandit, and a 5x5 gridworld navigation task.
Zero-Shot Regression Performance
In the regression task, the Laplace methods were tested for their ability to capture underlying function distributions. As shown in Figure 1, the "Sum $\Sigma$" variant (Markovian) consistently tracks the performance of full-distribution baselines.
Crucially, in the "Post Hoc" scenario, the Laplace Sum $\Sigma$ method significantly outperforms the Stationary variant and remains competitive with models trained explicitly for variational inference. This demonstrates that deterministic RNNs often learn representations that contain sufficient information for Bayesian inference, even if they do not explicitly represent it.
Key Insights
- Overconfidence: Empirical results confirm that standard RNN-based agents are overconfident estimators that do not satisfy mathematical consistency in their internal beliefs.
- Efficiency: The Laplace VRNN achieves performance parity with standard VRNN baselines while being more parameter-efficient, as it derives variance from network sensitivity rather than learning additional output parameters.
- Stability: In RL tasks like the bandit and gridworld, Laplace-augmented agents achieved sub-linear cumulative regret, indicating effective online adaptation.
Conclusion
The Laplace VRNN provides a powerful, lightweight mechanism for adding uncertainty quantification to meta-RL agents. By leveraging the curvature of learned representations, it enables Bayesian benefits—such as entropy estimation and improved robustness—without the overhead of traditional variational training. While the Gaussian assumption and Jacobian computation present some limitations, the ability to apply this method post-hoc offers a significant tool for analyzing and improving existing meta-RL models.
Summary updated