TL;DR
Model-based reinforcement learning (RL) often fails in complex environments because agents waste their "mental capacity" trying to model every flickering leaf or passing car in the background. Task Informed Abstractions (TIA) solves this by splitting the world model into two parts: a "Task Model" that focuses on reward-relevant features and a "Distractor Model" that captures everything else. By using adversarial training to keep rewards out of the distractor model, TIA allows agents to ignore visual noise and achieve state-of-the-art performance on cluttered benchmarks.
What Problem Does This Paper Address?
In model-based RL, an agent learns a "world model" to predict the consequences of its actions. Standard approaches, like the popular Dreamer algorithm, typically use a reconstruction loss—meaning the model tries to recreate the input image pixel-by-pixel.
The problem is that reconstruction is "task-agnostic." In a self-driving simulation, the model might spend all its capacity perfectly predicting the movement of clouds or trees (distractors) while failing to precisely track the position of a pedestrian (task-relevant). As shown in the paper's introductory experiment, even a small model can solve a task with a clean background, but when complex video distractions are added, performance plummets because the model gets "distracted" by the background noise.
Methods
The authors introduce a new formalism called the Task Informed MDP (TiMDP). The core idea is to factor the state space into two independent components: $s^+$ (task-relevant) and $s^-$ (distractor).
To implement this, TIA trains two separate models in parallel:
- The Task Model: This model learns features ($s^+$) used to predict both the reward and the next state.
- The Distractor Model: This model learns features ($s^-$) that are explicitly prevented from knowing anything about the reward.
The "secret sauce" is how these models interact:
- Adversarial Dissociation: The distractor model is trained with an adversarial loss. A reward-prediction head tries to guess the reward from $s^-$, while the distractor model's encoder is optimized to make that guess as difficult as possible. This ensures $s^-$ contains zero task-relevant information.
- Cooperative Reconstruction: Both models work together to reconstruct the original observation. They use a learned "mask" to decide which model is responsible for which pixels. This allows the distractor model to "explain away" the background, leaving the task model to focus on the agent and its goal.
Datasets
The authors evaluated TIA across three increasingly difficult domains:
- ManyWorld: A custom environment where a blue block must reach a red goal while surrounded by visually similar colored blocks acting as distractors.
- DMC with Natural Video Distraction: Standard DeepMind Control tasks (Cheetah, Walker, Hopper) where the background is replaced with grayscale videos of driving cars from the Kinetics dataset.
- Atari (ALE): Challenging games like Robotank and Yars Revenge that contain innate visual clutter (radar scanners, UI elements, etc.) that are not directly related to the immediate reward.
Key Insights and Ideas
- Reconstruction captures too much, Rewards capture too little: Pure reconstruction models get lost in the noise, while models trained only on rewards often lack the rich features needed for complex control. TIA finds the middle ground by using reconstruction to learn features but segregating them based on reward relevance.
- The "Explaining Away" Effect: By giving the distractor model the specific job of modeling the background, the task model naturally gravitates toward the agent's body and the goal objects. This division of labor results in cleaner, more focused representations for the policy to act upon.
- Factored Dynamics: TIA assumes that the agent and the background evolve independently. While this is a simplification, it proves to be a powerful inductive bias for learning in cluttered scenes.
Key Results
TIA consistently outperformed state-of-the-art baselines like Dreamer, DeepMDP, and DBC (Bisimulation):
- Robustness to Noise: In the Cheetah Run task with video backgrounds, TIA achieved nearly double the episode return of the standard Dreamer algorithm.
- Sample Efficiency in Atari: On Robotank, TIA reached higher scores at 100 million steps than standard model-free algorithms (DQN/Rainbow) reached at 200 million steps.
- Qualitative Clarity: Visualizations of the learned masks show that TIA's task model successfully ignores rotating radars and textures in Robotank, focusing instead on the crosshairs and enemy tanks.
Limitations and Future Work
- Hyperparameter Tuning: The balance between the adversarial reward loss and the reconstruction loss ($\lambda_{Radv}$ and $\lambda_{Os}$) is currently domain-dependent. The authors suggest that future work could automate this tuning.
- Interaction Violations: The "factored dynamics" assumption fails if a distractor physically interacts with the task-relevant objects (e.g., a distractor block colliding with the agent). TIA currently handles this by treating strong interactions as "task-relevant," but more complex interactions might require a more flexible model.
- Sparse Rewards: The adversarial separation relies on a reward signal. In environments where rewards are extremely rare, the model might struggle to distinguish between task and distractor features early in training.
Summary updated