Introduction
The emergence of Robotics Transformers (RT) has significantly advanced the reasoning capabilities of robotic systems by utilizing large-scale models pre-trained on diverse internet data. However, these models are often constrained by the "Transformer tax"—the quadratic computational complexity ($O(L^2)$) relative to sequence length ($L$). For real-time robotic applications involving high-resolution imagery or long action histories, this complexity leads to high latency, making deployment impractical. While linear attention mechanisms ($O(L)$) offer a potential solution, they frequently suffer from performance degradation compared to their quadratic counterparts. This paper introduces SARA-RT (Self-Adaptive Robust Attention), a method to convert large, pre-trained Transformers into efficient linear-attention models without sacrificing accuracy.
Methodology: SARA Mapping and Up-training
The core of the proposed approach is the SARA Mapping and a novel up-training paradigm.
SARA Mapping
Standard Transformers rely on softmax attention, which is computationally expensive. SARA-RT replaces this with a kernel-based linear attention mechanism: $$\phi(z) = v \odot f(Gz)$$ where $G$ is a learnable matrix that enables the model to "self-adapt" its attention patterns. This learnable pre-processing step allows the model to approximate the "spiky" attention distributions of softmax, which are essential for focusing on specific objects within a robotic scene.
Up-training
Instead of training a linear model from scratch, the authors employ "up-training." This involves taking a high-performing, pre-trained quadratic model (such as the 5B-parameter RT-2) and replacing its attention layers with SARA layers. The model is then fine-tuned, allowing it to inherit the complex reasoning capabilities of the original model while gaining the efficiency of linear attention.
Experimental Setup and Datasets
SARA-RT was evaluated across multiple domains to test its robustness and generalization:
- Matterport: Zero-shot VR navigation tasks.
- Point Cloud Grasping: Simulated environments requiring 3D point cloud processing for object manipulation.
- RT-2 Robotics Data: A large-scale dataset of expert demonstrations covering tasks like picking, knocking, and drawer manipulation.
- Real-World Evaluation: Physical robot testing across seven distinct manipulation tasks.
Key Results and Discussion
The implementation of SARA-RT yielded significant improvements in both efficiency and performance.
Efficiency Gains
For the 5B-parameter RT-2 model, SARA-RT achieved a 14% speedup on TPU. The benefits were even more pronounced in Point Cloud Transformers (PCT), where SARA-PCT maintained a constant inference time (~100ms) regardless of point cloud density, whereas standard PCT latency increased linearly with the number of points.
Performance Improvements
Surprisingly, SARA-RT did not just match the baseline; it often exceeded it. The best SARA-RT variant showed a 12% improvement in mean accuracy over the original RT-2. In real-world physical tests, SARA-RT-2 achieved 100% accuracy on specific "pick" tasks where the baseline was less reliable.
Conclusion and Future Work
SARA-RT demonstrates that linear attention, when combined with up-training and learnable kernels, can provide the speed necessary for real-time robotics without compromising reasoning quality. Future work includes applying SARA to the entire model pipeline (including the vision-text fuser) and exploring the use of much higher resolution images for more precise robotic control.
Summary updated