SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention

Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael S. Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamás Sarlós, Ken Oslund, Karol Hausman, Kanishka Rao

TL;DR

SARA-RT introduces a novel "up-training" method that converts computationally expensive, pre-trained Robotics Transformers (like the 5B-parameter RT-2) into highly efficient linear-attention models. This approach significantly reduces latency and memory usage while maintaining—and in some cases, even improving—the robot's task performance.

What Problem Does This Paper Address?

The "Robotics Transformer" (RT) revolution has brought incredible reasoning capabilities to robots by leveraging massive models pre-trained on internet-scale data. However, these models come with a heavy "Transformer tax": quadratic complexity.

In standard Transformers, the computational cost grows quadratically ($O(L^2)$) with the sequence length ($L$). For a robot, the sequence length might represent high-resolution images, long histories of past actions, or dense 3D point clouds. This makes large models like RT-2 prohibitively slow for real-time deployment (often running at only a few Hertz), which is dangerous or impractical for fluid robotic manipulation. While "linear attention" alternatives exist to reduce complexity to $O(L)$, they typically require training from scratch and often fail to match the performance of their quadratic counterparts.

Methods

The authors propose SARA-RT (Self-Adaptive Robust Attention) and a training paradigm called up-training.

  1. SARA Mapping: Instead of using the standard softmax attention, they use a kernel-based linear attention mechanism. The key innovation is a learnable pre-processing step: $\phi(z) = v \odot f(Gz)$. Here, $G$ is a learnable matrix that allows the model to "self-adapt" its attention patterns.
  2. Up-training: Rather than training a linear model from scratch, the researchers take a high-performing, pre-trained quadratic model (like RT-2) and replace its attention layers with SARA layers. They then fine-tune the model. This allows the linear model to "inherit" the complex reasoning capabilities of the original model while learning to be efficient.
  3. Mathematical Approximation: The paper provides a rigorous proof showing that with the right parameters, SARA can accurately approximate the "spiky" attention distributions of regular softmax attention, which is crucial for focusing on specific objects in a scene.

Datasets

The researchers evaluated SARA-RT across several domains:

  • Matterport: Used for zero-shot VR navigation tasks to build initial intuition.
  • Point Cloud Grasping: A simulated environment where the robot must grasp objects (coke cans, bananas, etc.) based on 3D point cloud data.
  • RT-2 Robotics Data: A massive dataset of expert robotic demonstrations (the same used for the original RT-1 and RT-2 models) covering tasks like picking, knocking, and drawer manipulation.
  • Real-World Evaluation: The models were tested on physical robots across seven distinct manipulation tasks, including a "diverse pick" task to test generalization.

Key Insights and Ideas

  • Linear doesn't have to mean "worse": The common wisdom is that linear attention is a compromise. SARA-RT shows that with "up-training," you can get the speed of linear attention without the accuracy drop.
  • The Power of Learnable Kernels: By making the projection matrices ($G$) learnable, the model can recreate the "spiky" attention patterns that quadratic models use to focus on small, important details (like the handle of a drawer).
  • Action Representation Matters: The authors found that representing robot actions as continuous vectors (rounded strings) rather than discrete tokens led to higher quality models when combined with their architecture.
  • Scaling Point Clouds: For Point Cloud Transformers (PCT), SARA-RT provides constant inference time regardless of the number of points (up to 5,000 tested), whereas regular PCT slows down significantly as the scene gets denser.

Key Results

  • RT-2 Speedup: SARA-RT achieved a 14% speedup on TPU for the 5B-parameter RT-2 model. On CPU, the speedup for the ViT encoder was even more dramatic, especially at higher image resolutions.
  • Performance Boost: Surprisingly, the best SARA-RT variant (using 3-frame history and vector action representation) provided a 12%+ improvement in mean accuracy over the baseline RT-2.
  • Efficiency in Point Clouds: In grasping tasks, SARA-PCT maintained a constant ~100ms inference time, while the regular PCT's latency spiked to nearly 1 second as the point cloud size increased.
  • Real-World Success: In physical robot tests, SARA-RT-2 achieved 100% accuracy on certain "pick" tasks where the original model was less reliable.

Limitations and Future Work

  • Partial Implementation: In the RT-2 experiments, the authors only applied SARA to the ViT encoder. The "fuser" (the part that combines vision and text) still used regular attention. Applying SARA to the entire pipeline is a logical next step.
  • Computational Trade-offs: While the linear attention is faster, the "vector representation" and "history" versions of the model are more memory-intensive, which might be a challenge for some edge deployment scenarios.
  • High-Resolution Potential: The authors suggest that SARA-RT opens the door to using much higher resolution images for robotic control—something currently impossible with quadratic Transformers—but they haven't fully explored this frontier yet.

Summary updated

Abstract

We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (including massive billion-parameter vision-language-action models or VLAs), into their efficient linear-attention counterparts maintaining high quality. We demonstrate the effectiveness of SARA-RT by speeding up: (a) the class of recently introduced RT-2 models [1], the first VLA robotic policies pre-trained on internet-scale data, as well as (b) Point Cloud Transformer (PCT) robotic policies operating on large point clouds. We complement our results with the rigorous mathematical analysis providing deeper insight into the phenomenon of SARA.
BibTeX
@inproceedings{leal2024sara,
  author = {Isabel Leal and Krzysztof Choromanski and Deepali Jain and Avinava Dubey and Jake Varley and Michael S. Ryoo and Yao Lu and Frederick Liu and Vikas Sindhwani and Quan Vuong and Tamás Sarlós and Ken Oslund and Karol Hausman and Kanishka Rao},
  title = {{SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=k3RRsew98x},
}
Figure