TL;DR
CtrlFormer is a novel Transformer-based framework that learns transferable state representations for visual reinforcement learning (RL). By introducing task-specific "policy tokens" and a contrastive co-training objective, it achieves state-of-the-art sample efficiency and successfully transfers knowledge across different control tasks without the "catastrophic forgetting" that plagues traditional methods.
Introduction
In the world of visual control—where agents learn to act based on raw pixels—two major hurdles persist: sample efficiency and transferability.
Reinforcement learning is notoriously data-hungry, often requiring millions of interactions to learn simple tasks. While we’d love to "transfer" knowledge from one task (like balancing a pole) to another (like swinging it up), most models either fail to adapt quickly or suffer from catastrophic forgetting, where learning a new skill causes them to completely lose the old one.
Existing solutions like pre-training on ImageNet often suffer from a "domain gap" (robotic arms don't look like internet photos), and modular approaches like Progressive Neural Networks don't scale well as the number of tasks grows. CtrlFormer aims to solve this by leveraging the inherent flexibility of Transformers to model the relationships between different tasks and visual observations.
Methods
The core of CtrlFormer is a pyramidal Vision Transformer (ViT) architecture designed specifically for the constraints of RL.
- Policy Tokens: Instead of a single global representation, CtrlFormer assigns a unique, learnable "policy token" to each task. During the self-attention process, these tokens act as queries that "attend" to relevant image patches and even to other policy tokens. This allows the model to decouple task-specific logic from general visual features.
- Contrastive Co-training: Transformers are famously data-hungry. To make them viable for RL, the authors use a self-supervised contrastive loss (inspired by BYOL) as an auxiliary task. This provides a dense training signal that helps the model learn robust visual representations even when RL rewards are sparse.
- Pyramidal Structure: To manage computational costs, the model uses a three-stage pyramid structure that progressively reduces the number of visual tokens through pooling, making it more efficient than a standard ViT.
- Transfer Mechanism: When a new task is introduced, the model simply adds a new policy token. The shared backbone weights are fine-tuned, allowing the new task to "borrow" visual features learned from previous tasks.
Datasets
The authors evaluated CtrlFormer using the DeepMind Control (DMControl) suite, a standard benchmark for continuous control. They focused on four domains:
- Walker: Stand and Walk.
- Cartpole: Swingup and Swingup-sparse.
- Reacher: Easy and Hard.
- Finger: Turn-easy and Turn-hard.
They also tested on the DeepMind Manipulation benchmark, which involves a Kinova robotic arm performing object-reaching tasks.
Key Insights and Results
- Decoupling is Key: By using separate policy tokens, the model can maintain high performance on old tasks while learning new ones. The self-attention mechanism naturally decides which parts of the "old" knowledge are relevant to the "new" task.
- Attention as a Filter: CtrlFormer learns to ignore background noise and focus strictly on task-relevant objects (like the tip of a finger or the joint of a pole), whereas standard ResNets are often "distracted" by irrelevant pixels.
- The Power of Transfer: The model doesn't just avoid forgetting; it actually learns faster. Transferring from an "easy" task to a "hard" task in the same domain provides a massive boost in sample efficiency.
- Zero to Hero: In the "Cartpole swingup-sparse" task, standard state-of-the-art methods (like DrQ) failed completely, scoring 0 after 100k samples. CtrlFormer, leveraging transfer from a previous task, achieved a score of 769±34 with the same amount of data.
- No Forgetting: After learning a second task, CtrlFormer’s performance on the first task remained stable (e.g., 842 vs. 846 in Cartpole), while the baseline DrQ’s performance dropped by more than half.
- Cross-Domain Success: The model successfully transferred knowledge even between different domains, such as moving from the "Finger" domain to the "Reacher" domain, outperforming CNN-based baselines.
Limitations and Future Work
While CtrlFormer is a significant step forward, the authors note a few areas for improvement:
- Computational Overhead: Although the pyramidal structure helps, Transformers are still more computationally intensive than simple CNNs.
- Sequential Saturation: The paper primarily explores transferring between two or four tasks. It remains to be seen how the shared backbone holds up when scaled to dozens or hundreds of diverse tasks.
- Future Directions: The authors suggest exploring more advanced Transformer variants and applying this framework to even more complex, long-horizon manipulation tasks.
Summary updated