Abstract
Current model-based reinforcement learning methods struggle when operating from complex visual scenes due to their inability to prioritize task-relevant features. To mitigate this problem, we propose learning Task Informed Abstractions (TIA) that explicitly separates reward-correlated visual features from distractors. For learning TIA, we introduce the formalism of Task Informed MDP (TiMDP) that is realized by training two models that learn visual features via cooperative reconstruction, but one model is adversarially dissociated from the reward signal. Empirical evaluation shows that TIA leads to significant performance gains over state-of-the-art methods on many visual control tasks where natural and unconstrained visual distractions pose a formidable challenge.
Introduction: The Reconstruction Bottleneck
In model-based reinforcement learning (MBRL), an agent learns a "world model" to predict the consequences of its actions. A popular approach is to train this model by forcing it to reconstruct the visual input (pixels). However, in visually complex scenes—like a robot walking through a forest—most pixels (swaying leaves, moving clouds) are completely irrelevant to the task.
Standard MBRL agents like Dreamer try to model every single pixel. Because model capacity is finite, the agent often wastes its "brainpower" representing the complex background while failing to capture the fine details of its own body or the goal. This leads to poor performance in environments with "natural" distractions, as model capacity becomes a zero-sum game.
Methods: Task Informed Abstractions (TIA)
The authors introduce a new formalism called the Task Informed MDP (TiMDP) and a corresponding architecture, TIA. The core idea is to factor the latent state space into two distinct components:
- Task Model ($s^+_t$): Responsible for capturing features that correlate with the reward signal and predicting future task-relevant states.
- Distractor Model ($s^-_t$): Responsible for capturing everything else needed to reconstruct the image but which has no bearing on the reward.
Architecture and Training
The TIA architecture employs two key mechanisms to ensure effective separation:
- Cooperative Reconstruction: Both models' decoders work together to reconstruct the original observation. They use a Learned Mixing Mask to decide which model "owns" which pixels. This forces the models to partition the scene, effectively "explaining away" the background.
- Adversarial Dissociation: To ensure the distractor model doesn't accidentally learn task-relevant info, the authors use an adversarial loss. A reward predictor tries to guess the reward from the distractor features, while the distractor model is trained to make that prediction as difficult as possible.
- Focused Imagination: When training the agent's policy, it only "imagines" future trajectories using the Task Model. The distractors are effectively filtered out.
Key Results and Insights
The researchers evaluated TIA across diverse domains, including DeepMind Control (DMC) with natural video backgrounds and cluttered Atari games.
- Superior Performance: On DMC tasks with video backgrounds, TIA significantly outperformed Dreamer, DBC (Deep Bisimulation for Control), and DeepMDP, often achieving 2-3x higher episode returns.
- Robustness to Model Size: TIA maintains high performance even when the model size is reduced, whereas standard methods fail because they cannot fit both the task and the background into a limited latent space.
- Qualitative Success: Visualizations of the TIA mask show it successfully isolating the agent and goal while relegating the moving video background to the distractor model.
- Sample Efficiency: On Atari games like Robotank, TIA matched or beat the performance of model-free baselines (like Rainbow DQN) while using significantly fewer environment steps.
Limitations and Future Work
While TIA provides a robust way to handle distractions, it relies on a "factored dynamics" assumption—that task-relevant objects and distractors evolve independently. This assumption might break down if a distractor strongly interacts with a task object (e.g., a collision). Additionally, while TIA handles distractions seen during training, it does not automatically ignore entirely new types of distractions at test time. Future work may incorporate attention mechanisms to improve out-of-distribution generalization.
Summary updated