CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer

Yao Mu, Shoufa Chen, Mingyu Ding, Jianyu Chen, Runjian Chen, Ping Luo

Abstract

Transformer has achieved great successes in learning vision and language representation, which is general across various downstream tasks. In visual control, learning transferable state representation that can transfer between different control tasks is important to reduce the training sample size. However, porting Transformer to sample-efficient visual control remains a challenging and unsolved problem. To this end, we propose a novel Control Transformer (CtrlFormer), possessing many appealing benefits that prior arts do not have. Firstly, CtrlFormer jointly learns self-attention mechanisms between visual tokens and policy tokens among different control tasks, where multitask representation can be learned and transferred without catastrophic forgetting. Secondly, we carefully design a contrastive reinforcement learning paradigm to train CtrlFormer, enabling it to achieve high sample efficiency, which is important in control problems. For example, in the DMControl benchmark, unlike recent advanced methods that failed by producing a zero score in the "Cartpole" task after transfer learning with 100k samples, CtrlFormer can achieve a state-of-the-art score with only 100k samples while maintaining the performance of previous tasks.

Introduction

In visual reinforcement learning (RL), agents must learn to map high-dimensional raw pixels to continuous actions. This dual challenge involves learning a robust visual representation of the environment while simultaneously optimizing a complex control policy. While Transformers have revolutionized fields like computer vision and NLP due to their generalizability, they are typically "data-hungry," requiring massive datasets that are often unavailable or expensive to collect in RL settings.

Furthermore, traditional neural networks in RL suffer from catastrophic forgetting. When an agent is trained on a new task (e.g., transitioning from walking to running), the new weight updates often overwrite the knowledge required for previous tasks. The goal of CtrlFormer is to provide a sample-efficient architecture that enables seamless knowledge transfer across tasks without sacrificing performance on earlier learned behaviors.

The CtrlFormer Framework

CtrlFormer introduces several architectural innovations designed to handle the specific constraints of visual control:

  1. Policy Tokens: Similar to the [CLS] token in Vision Transformers (ViT), CtrlFormer assigns a unique, learnable "policy token" to every task. These tokens interact with visual "patch tokens" through self-attention, allowing the model to decouple task-specific behaviors from general visual features.
  2. Contrastive Reinforcement Learning: To address the sample-inefficiency of standard Transformers, CtrlFormer employs a co-training paradigm. It combines a standard RL loss (using Soft Actor-Critic, SAC) with a self-supervised contrastive loss. This auxiliary task forces the encoder to learn meaningful visual representations from significantly fewer samples.
  3. Pyramidal Structure: The architecture utilizes a pyramidal design where visual tokens are pooled across three stages to manage computational complexity, while policy tokens are preserved throughout to maintain task-specific information.
Overview of the CtrlFormer architecture showing the interaction between visual patch tokens and task-specific policy tokens.

When transitioning to a new task, a new policy token is initialized. Because the visual encoder and existing policy tokens are already pre-trained, the agent can leverage this "warm" state to learn the new task rapidly.

Experimental Evaluation

The model was evaluated on the DeepMind Control Suite (DMControl), a standard benchmark for continuous control. The experiments covered both same-domain and cross-domain transfers across several environments, including Walker, Cartpole, Reacher, Finger, and a Kinova robotic arm manipulation task.

Key Insights

  • Attention as Task Relevancy: Self-attention allows the model to automatically discover relationships between tasks. If a new task is similar to a previously learned one, the new policy token will attend more strongly to the representations of the relevant old token.
  • Decoupling for Maintainability: By using separate tokens for different tasks while sharing a visual backbone, CtrlFormer avoids catastrophic forgetting. Retesting on an old task simply requires swapping back to its original policy token.
  • Visual Grounding: Visualization via Grad-CAM showed that CtrlFormer focuses on task-relevant objects (e.g., the tip of a finger), whereas baselines like ResNet-based agents are often distracted by background noise.

Results and Discussion

CtrlFormer demonstrates superior performance in both maintainability and transferability compared to advanced baselines like DrQ and Dreamer.

Comparison of maintainability and transferability between CtrlFormer and the DrQ baseline across different DMControl tasks.

Superior Transferability

In the Finger (turn-hard) task, transferring knowledge allowed CtrlFormer to reach a score of approximately 569, significantly higher than the ~344 achieved when learning from scratch. Most notably, in the difficult "Cartpole (sparse)" task, recent methods like DrQ failed completely (scoring zero) after transfer learning with 100k samples. In contrast, CtrlFormer achieved a state-of-the-art score of 769±34 with the same sample budget.

Maintainability

Unlike traditional methods where performance on Task A drops after learning Task B, CtrlFormer maintains near-original performance levels on Task A. This is evidenced by the "Retest" scores in the maintainability benchmarks, where CtrlFormer's performance remains stable while baselines show significant degradation after new task training.

Conclusion

CtrlFormer represents a significant step forward in applying Transformers to sample-efficient visual control. By leveraging policy tokens and contrastive learning, it achieves state-of-the-art transferability and maintainability. While the linear growth of policy tokens with the number of tasks and hyperparameter sensitivity remain areas for future optimization, the framework provides a robust foundation for general-purpose visual control agents.

Summary updated

Abstract

Transformer has achieved great successes in learning vision and language representation, which is general across various downstream tasks. In visual control, learning transferable state representation that can transfer between different control tasks is important to reduce the training sample size. However, porting Transformer to sample-efficient visual control remains a challenging and unsolved problem. To this end, we propose a novel Control Transformer (CtrlFormer), possessing many appealing benefits that prior arts do not have. Firstly, CtrlFormer jointly learns self-attention mechanisms between visual tokens and policy tokens among different control tasks, where multitask representation can be learned and transferred without catastrophic forgetting. Secondly, we carefully design a contrastive reinforcement learning paradigm to train CtrlFormer, enabling it to achieve high sample efficiency, which is important in control problems. For example, in the DMControl benchmark, unlike recent advanced methods that failed by producing a zero score in the "Cartpole" task after transfer learning with 100k samples, CtrlFormer can achieve a state-of-the-art score with only 100k samples while maintaining the performance of previous tasks. The code and models are released in our project homepage.
BibTeX
@inproceedings{mu2022ctrlformer,
  author = {Yao Mu and Shoufa Chen and Mingyu Ding and Jianyu Chen and Runjian Chen and Ping Luo},
  title = {{CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=apQ1_Zggyb},
}
Figure