TL;DR
The Attend, Adapt and Transfer (A2T) framework introduces an attentive deep architecture for reinforcement learning that selectively pulls knowledge from multiple source experts. By using a state-dependent attention mechanism and a "base network" that learns from scratch, A2T effectively avoids negative transfer (where bad advice slows down learning) while successfully combining the strengths of different experts across different parts of a task.
What Problem Does This Paper Address?
Transfer learning—applying knowledge from one task to a new one—is a "holy grail" in Artificial Intelligence. In Reinforcement Learning (RL), however, it faces two major hurdles:
- Negative Transfer: If a source task is too different from the target, trying to use its knowledge can actually make the agent perform worse than if it had started from scratch.
- Selective Transfer: Real-world tasks are complex. A robot might need "Expert A" to navigate a hallway but "Expert B" to open a door. Most transfer methods struggle to switch between multiple sources or identify which source is relevant for a specific part of the environment (the state space).
The authors aim to create an agent that can "attend" to the right expert at the right time, "adapt" that knowledge, and "transfer" it without getting tripped up by irrelevant or harmful data.
Methods
The core of the paper is the A2T Architecture.
It consists of three main components:
- Source Experts ($K_1 \dots K_N$): Pre-trained models from previous tasks. Crucially, these remain fixed to prevent "unlearning" their original skills.
- Base Network ($K_B$): A network that learns the target task from scratch. This acts as a "safety valve"—if no experts are helpful, the system can rely on the base network to avoid negative transfer.
- Attention Network: This is the "brain" of the operation. For any given state (e.g., a specific frame in a video game), it outputs a set of weights. These weights determine how much the agent should listen to each expert versus the base network.
The final output is a convex combination (a weighted average) of all these networks. The system uses "soft attention," meaning it can blend advice from multiple experts simultaneously. The authors demonstrate that this framework is generic: it works for Policy Transfer (transferring how to act) and Value Transfer (transferring the estimated worth of actions, as in Deep Q-Networks).
Datasets
The researchers evaluated A2T across several simulated environments:
- Chain World & Puddle World: Simple grid-like environments used to test basic policy transfer and the ability to navigate toward goals.
- Atari 2600 (Pong & Breakout): Used for complex value transfer.
- Synthetic "Bad" Experts: To test negative transfer, they created Inverse-Pong (where the reward is flipped, so the expert tries to lose) and used Freeway (a completely different game) as a source for Pong.
- Blurred Experts: They created specialized experts by blurring parts of the screen (e.g., an expert that only knows how to hit the ball in the top half of the screen).
Key Insights and Ideas
- The "Safety Valve" Concept: By including a base network that learns from scratch, the attention mechanism can simply "turn off" the experts if they are providing harmful advice. This is the primary mechanism for avoiding negative transfer.
- State-Level Granularity: Unlike many methods that decide to transfer an entire task or nothing at all, A2T makes decisions at the level of individual states. It can use Expert A for the left side of the screen and Expert B for the right.
- Learning by Replicating: Interestingly, the base network doesn't just learn from the environment; it also learns to replicate the behavior of useful experts. This helps the base network "catch up" and eventually fine-tune those skills specifically for the target task.
Key Results
- Selective Transfer in Pong: When given two "partial" experts (one for the top half of the screen, one for the bottom), A2T learned to switch between them perfectly. It achieved a score of 17.2 after just one epoch, whereas learning from scratch would take much longer.
- Defeating Negative Transfer: When faced with "Inverse-Pong" (the "bad" expert), A2T's performance didn't drop. The attention network quickly learned to assign a weight of nearly zero to the bad expert and relied on the base network instead.
- Breakout Success: In Breakout, A2T achieved a score of 94.5 by successfully attending to experts trained on different quadrants of the screen, significantly outperforming agents that didn't use the attention mechanism.
Limitations and Future Work
- Same-Domain Constraint: Currently, A2T requires the source and target tasks to share the same state and action spaces (e.g., the same screen resolution and the same joystick inputs).
- Fixed Experts: While keeping experts fixed prevents unlearning, it also means the experts can't "evolve" alongside the target task.
- Future Directions: The authors suggest extending this to Hierarchical RL, where the attention network acts as a "meta-controller" selecting between different sub-policies or "skills" to solve even more complex, multi-stage problems.
Summary updated