RL$^3$: Boosting Meta Reinforcement Learning via RL inside RL$^2$

Abhinav Bhatia, Samer Nashed, Shlomo Zilberstein

TL;DR

This paper introduces RL$^3$, a novel meta-reinforcement learning (Meta-RL) approach that injects traditional RL's learned value functions into Meta-RL agents. This hybrid method significantly boosts performance on long-horizon and out-of-distribution tasks, drastically reduces meta-training time, and offers better generalization compared to existing Meta-RL techniques like RL$^2$.

What Problem Does This Paper Address?

Meta-RL aims to train agents that can quickly adapt to new tasks within a distribution. Existing methods, like RL$^2$, often rely on sequence models (RNNs, Transformers) to process entire experience histories. While this allows for data-efficient learning, it struggles with long-horizon tasks where the history becomes too long for effective reasoning, and with out-of-distribution (OOD) tasks that deviate from the training distribution. Traditional RL, on the other hand, is data-inefficient but converges to optimal policies with enough data. The core challenge is to combine the fast adaptation of Meta-RL with the asymptotic performance and robustness of traditional RL, especially for complex, long-horizon, and unseen tasks.

Methods

The authors propose RL$^3$, a hybrid approach that augments the input to the Meta-RL agent with task-specific Q-value estimates. These Q-values are learned using traditional, object-level RL algorithms (like Q-learning) for each task. This "value-augmented MDP" (VAMDP) then serves as the input to a Meta-RL algorithm, which in this paper is an enhanced version of RL$^2$ using transformers. The key idea is that Q-values, being general-purpose RL components, can effectively summarize experience histories, provide direct actionability, and offer a more robust signal for task identification and adaptation, especially when dealing with long-term dependencies and OOD scenarios. The authors also explore a "RL$^3$-coarse" variant that uses abstract states for Q-value estimation, demonstrating computational efficiency.

Datasets

The experiments were conducted on:

  • Bernoulli Multi-Armed Bandits: A standard benchmark for Meta-RL, used here as a sanity check.
  • Random MDPs: A benchmark from Duan et al. (2016) with varying reward and transition dynamics.
  • GridWorld Navigation: Custom discrete domains (11x11 and 13x13 grids) designed to exhibit short-term, long-term, and complex dependencies, with variations for testing OOD generalization (e.g., DENSE, DETERMINISTIC, WATERY, DANGEROUS, CORNER).

Key Insights and Ideas

  • Q-values as powerful experience summarizers: Injecting Q-value estimates into Meta-RL agents acts as a form of compression, distilling complex, long histories into a more manageable and informative representation. This allows the meta-learner to focus on higher-level decision-making.
  • Bridging the gap between Meta-RL and traditional RL: RL$^3$ effectively combines the strengths of both paradigms. It leverages the data efficiency and adaptation capabilities of Meta-RL while incorporating the asymptotic optimality and robustness of traditional RL through Q-value estimates.
  • Improved OOD generalization: The Q-value estimates provide a more generalizable signal for task identification, making RL$^3$ less susceptible to distribution shifts compared to pure sequence-model-based Meta-RL.
  • Faster meta-training: By providing more informative inputs, RL$^3$ significantly reduces the number of meta-training iterations required to achieve high performance.
  • Actionability of Q-values: Q-estimates directly inform optimal exploitation, allowing the meta-agent to learn to rely on them, especially in the later stages of adaptation, effectively "ignoring" the raw history when sufficient Q-value information is available.

Key Results

  • Performance Boost: RL$^3$ consistently matched or exceeded RL$^2$-transformer performance across all tested domains.
  • OOD Generalization: RL$^3$ demonstrated superior OOD generalization, particularly in the MDPs and GridWorlds domains.
  • Long-Horizon Advantage: The benefits of RL$^3$ increased with longer interaction periods, where RL$^2$-transformer performance sometimes degraded.
  • Meta-Training Speedup: RL$^3$ achieved comparable performance to RL$^2$-transformer in a fraction of the meta-training time (e.g., requiring only 50% and 30% of iterations for 11x11 and 13x13 GridWorlds, respectively).
  • RL$^3$-coarse Efficiency: The coarse-grained version of RL$^3$ achieved excellent performance with significantly reduced computational overhead (around 10% overhead) while still outperforming RL$^2$-transformer.

Limitations and Future Work

  • MDP Assumption: RL$^3$ currently assumes object-level decision-making is an MDP. Extending it to POMDPs using methods like point-based value iteration is a potential future direction.
  • Continuous State Spaces: The current object-level RL implementation (value iteration) is primarily for discrete states. Adapting it for continuous state spaces (e.g., using linear function approximation) is an area for future research.
  • Inference Time: While meta-training is faster, inference time at deployment is slightly slower due to running the object-level RL.
  • Task Identification: While Q-estimates provide strong signals for task identification, the paper suggests further exploration into how the Meta-RL agent can actively disambiguate trajectories to accelerate this process.

Summary updated

Abstract

Meta reinforcement learning (meta-RL) methods such as RL$^2$ have emerged as promising approaches for learning data-efficient RL algorithms tailored to a given task distribution. However, these RL algorithms struggle with long-horizon tasks and out-of-distribution tasks since they rely on recurrent neural networks to process the sequence of experiences instead of summarizing them into general RL components such as value functions. Moreover, even transformers have a practical limit to the length of histories they can efficiently reason about before training and inference costs become prohibitive. In contrast, traditional RL algorithms are data-inefficient since they do not leverage domain knowledge, but they do converge to an optimal policy as more data becomes available. In this paper, we propose RL$^3$, a principled hybrid approach that combines traditional RL and meta-RL by incorporating task-specific action-values learned through traditional RL as an input to the meta-RL neural network. We show that RL$^3$ earns greater cumulative reward on long-horizon and out-of-distribution tasks compared to RL$^2$, while maintaining the efficiency of the latter in the short term. Experiments are conducted on both custom and benchmark discrete domains from the meta-RL literature that exhibit a range of short-term, long-term, and complex dependencies.
BibTeX
@inproceedings{bhatia2025rl,
  author = {Abhinav Bhatia and Samer Nashed and Shlomo Zilberstein},
  title = {{RL$^3$: Boosting Meta Reinforcement Learning via RL inside RL$^2$}},
  booktitle = {RLC},
  year = {2025},
  url = {https://openreview.net/forum?id=dXyETfoeWc},
}
Figure