Abstract
Model-Based Reinforcement Learning (MBRL) provides a pathway to sample efficiency by learning environment dynamics, yet it often fails to fully exploit multitask information or maintain stability in "jittery" environments. We introduce TAPIG (Task Addressing for Prioritized Implicit Generalization), a data-centric approach that frames agent training as a meta-Markov Decision Process. By utilizing an attention-based meta-agent to selectively prioritize training data, TAPIG achieves superior sample efficiency, enhanced training stability, and improved generalization across varying tasks without requiring changes to the underlying model architecture.
Introduction
Deep Reinforcement Learning (RL) is notoriously sample-hungry and prone to overfitting. While Model-Based RL (MBRL) agents learn a "world model" to simulate dynamics, they often lack a principled mechanism to prioritize which experiences are most beneficial for learning a generalizable model. In environments with high parametric variability or instability—such as a robotic arm interacting with objects at varying angles—standard uniform sampling often leads to volatile performance.
Methods: The TAPIG Framework
The core innovation of TAPIG is framing the training process as a meta-Markov Decision Process (meta-MDP). In this setup, the "state" represents the current configuration of the RL agent and its training buffer, while an "action" corresponds to the selection of a specific batch of training data. The "reward" is defined as the performance improvement of the internal agent.
To navigate this meta-MDP, TAPIG employs an attention-style mechanism:
- Query: The agent’s current trajectory.
- Key: Potential data samples from a multitask buffer.
- Mechanism: The model uses "hard attention" to compare the current trajectory against a pool of experiences, selecting the most relevant ones to update the world model.
The addressing model is optimized using policy gradients (REINFORCE) to maximize internal agent rewards. The system is built upon Dreamer, utilizing its Recurrent State-Space Model (RSSM) for latent imagination.
Experimental Evaluation
The researchers evaluated TAPIG across two primary benchmarks:
- DeepMind Control Suite (DMC): Testing stability in the Hopper Hop, Finger Spin, and Acrobot Swingup environments.
- Rotated Drawer World (MetaWorld): Testing "implicit generalization" where an agent must learn to open a drawer across a full 360-degree range despite only seeing data from half the possible angles.
Key Results and Insights
TAPIG significantly reduced performance volatility in the DMC environments compared to the vanilla Dreamer baseline. Notably, it raised the performance "floor," ensuring that even the worst training runs maintained a higher level of success.
In the Rotated Drawer World, TAPIG maintained consistent performance across the entire parametric space, whereas Dreamer exhibited significant "blind spots" at specific angles. A key insight was that the meta-agent's success depends on a trained CNN for the addressing mechanism to identify specific reward-maximizing experiences rather than just abstract behavioral features.
Discussion and Limitations
TAPIG demonstrates that training can be treated as a controllable, optimized process rather than relying on simple heuristics like uniform sampling. However, the approach is sensitive to hyperparameters, particularly the "softmax temperature" controlling the sharpness of data selection. While TAPIG can bootstrap from its own data, its full potential is currently realized when paired with a multitask buffer. Future work may involve providing the addressing model with explicit task information to further enhance adaptability across even more diverse task sets.
Summary updated