SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention

Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael S. Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamás Sarlós, Ken Oslund, Karol Hausman, Kanishka Rao

Introduction

The emergence of Robotics Transformers (RT) has significantly advanced the reasoning capabilities of robotic systems by utilizing large-scale models pre-trained on diverse internet data. However, these models are often constrained by the "Transformer tax"—the quadratic computational complexity ($O(L^2)$) relative to sequence length ($L$). For real-time robotic applications involving high-resolution imagery or long action histories, this complexity leads to high latency, making deployment impractical. While linear attention mechanisms ($O(L)$) offer a potential solution, they frequently suffer from performance degradation compared to their quadratic counterparts. This paper introduces SARA-RT (Self-Adaptive Robust Attention), a method to convert large, pre-trained Transformers into efficient linear-attention models without sacrificing accuracy.

Methodology: SARA Mapping and Up-training

The core of the proposed approach is the SARA Mapping and a novel up-training paradigm.

SARA Mapping

Standard Transformers rely on softmax attention, which is computationally expensive. SARA-RT replaces this with a kernel-based linear attention mechanism: $$\phi(z) = v \odot f(Gz)$$ where $G$ is a learnable matrix that enables the model to "self-adapt" its attention patterns. This learnable pre-processing step allows the model to approximate the "spiky" attention distributions of softmax, which are essential for focusing on specific objects within a robotic scene.

Up-training

Instead of training a linear model from scratch, the authors employ "up-training." This involves taking a high-performing, pre-trained quadratic model (such as the 5B-parameter RT-2) and replacing its attention layers with SARA layers. The model is then fine-tuned, allowing it to inherit the complex reasoning capabilities of the original model while gaining the efficiency of linear attention.

Experimental Setup and Datasets

SARA-RT was evaluated across multiple domains to test its robustness and generalization:

  • Matterport: Zero-shot VR navigation tasks.
  • Point Cloud Grasping: Simulated environments requiring 3D point cloud processing for object manipulation.
  • RT-2 Robotics Data: A large-scale dataset of expert demonstrations covering tasks like picking, knocking, and drawer manipulation.
  • Real-World Evaluation: Physical robot testing across seven distinct manipulation tasks.
Physical robot setup for evaluating generalization in diverse picking tasks.

Key Results and Discussion

The implementation of SARA-RT yielded significant improvements in both efficiency and performance.

Efficiency Gains

For the 5B-parameter RT-2 model, SARA-RT achieved a 14% speedup on TPU. The benefits were even more pronounced in Point Cloud Transformers (PCT), where SARA-PCT maintained a constant inference time (~100ms) regardless of point cloud density, whereas standard PCT latency increased linearly with the number of points.

Performance Improvements

Surprisingly, SARA-RT did not just match the baseline; it often exceeded it. The best SARA-RT variant showed a 12% improvement in mean accuracy over the original RT-2. In real-world physical tests, SARA-RT-2 achieved 100% accuracy on specific "pick" tasks where the baseline was less reliable.

Robotic manipulation tasks evaluated using SARA-RT, including picking, knocking, and drawer manipulation.

Conclusion and Future Work

SARA-RT demonstrates that linear attention, when combined with up-training and learnable kernels, can provide the speed necessary for real-time robotics without compromising reasoning quality. Future work includes applying SARA to the entire model pipeline (including the vision-text fuser) and exploring the use of much higher resolution images for more precise robotic control.

Summary updated

BibTeX
@inproceedings{leal2024sara,
  author = {Isabel Leal and Krzysztof Choromanski and Deepali Jain and Avinava Dubey and Jake Varley and Michael S. Ryoo and Yao Lu and Frederick Liu and Vikas Sindhwani and Quan Vuong and Tamás Sarlós and Ken Oslund and Karol Hausman and Kanishka Rao},
  title = {{SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=k3RRsew98x},
}
Figure