RTPolicy: Real-Time (32.9 Hz) Multi-View Visuomotor Policy for Robotic Manipulation on Edge Devices

Xianbao Hou, Zehao Ni, Yonghao He, Lingfeng Qian, Xin Shi, Hu Su, Wei Sui, Cong YangarXiv 2026 · arXiv

TL;DR

RTPolicy is a high-efficiency visuomotor policy that brings complex, multi-view robotic manipulation to edge devices. By combining view-aware feature fusion with aggressive visual compression and a flow-matching action generator, it achieves state-of-the-art accuracy while running at real-time speeds (up to 32.9 Hz) on low-power hardware.

What Problem Does This Paper Address?

For a robot to perform precise tasks—like inserting a component into a slot or stacking blocks—it needs to "see" from multiple angles and react instantly to what it sees. This is known as closed-loop, multi-view manipulation.

However, there is a major bottleneck: latency. Current high-performing policies use massive visual encoders and process hundreds of "tokens" (data chunks representing parts of an image). On powerful GPUs, this is fine, but on edge devices (the small computers actually mounted on robots), it’s too slow. If a policy takes 500ms to decide on an action, the robot is essentially "driving blind" for half a second, leading to accumulated errors and failed tasks. The authors aim to bridge this gap, providing a policy that is both accurate and fast enough for real-time deployment on edge boards.

Methods

RTPolicy introduces a streamlined pipeline designed to minimize the computational load at every step:

  1. Lightweight Encoding: It uses a frozen DINOv3 ConvNeXt encoder. Keeping the encoder frozen saves training time, while the ConvNeXt architecture is more efficient for edge deployment than standard Transformers.
  2. View-Aware Feature Fusion: Instead of processing each camera view in isolation, RTPolicy adds learnable "view embeddings" to the features. It then uses a Transformer block to let tokens from different cameras "talk" to each other. This ensures the model understands the 3D relationship between views before any data is discarded.
  3. Visual Compression: This is the "efficiency engine." After fusion, the model uses residual downsampling and adaptive pooling to shrink the visual representation. On their real-world platform, they reduce 900 dense tokens down to just 12 compact tokens—a 75-fold reduction in the data the action generator has to process.
  4. Flow-Matching Action Generator: Instead of a traditional Diffusion Policy that might require 10-50 iterations to "denoise" an action, RTPolicy uses a Diffusion Transformer (DiT) with a flow-matching objective. This allows it to generate high-quality action chunks in just 4 integration steps, drastically cutting inference time.
The RTPolicy architecture highlights the path from multi-view images to compressed tokens and finally to action generation.

Datasets

The authors evaluated RTPolicy across both simulated and physical environments:

  • RoboTwin 2.0: A large-scale simulation benchmark for bimanual (two-armed) manipulation. They tested on the "clean2clean" single-task leaderboard across 50 diverse tasks.
  • Real-World Bimanual Platform: A custom setup using two ROKAE AR5 arms and three RealSense cameras (one head-mounted, two wrist-mounted). They designed three challenging tasks: Drag the Drawer, Stack and Cover, and Workpiece Assembly.

Key Insights and Ideas

  • Token Count is the Latency Killer: The paper demonstrates that the number of visual tokens entering the action generator is a primary driver of latency. By compressing these tokens after cross-view fusion, they maintain spatial awareness while slashing compute costs.
  • The "Streaming" Advantage: Low latency isn't just about speed; it's about accuracy. In their real-world tests, they found that running at 10 Hz (100ms updates) was significantly more successful than 2 Hz. Faster updates allow the robot to correct small deviations before they turn into task-ending failures.
  • Fusion Before Compression: A critical design choice was performing feature fusion before compression. This allows the model to retain complementary information from different views that might otherwise be lost if each view were compressed independently.

Key Results

RTPolicy consistently outperformed existing baselines in both speed and success rate:

  • Simulation: On the RoboTwin 2.0 benchmark, RTPolicy achieved a 52.26% mean success rate, surpassing the strongest baseline, $\pi_0$ (46.4%), and significantly beating Diffusion Policy (28.0%).
  • Real-World: In bimanual tasks, it reached a 72.0% mean success rate. For the most difficult task (Workpiece Assembly), it achieved 30% success, while most baselines failed completely or stayed near 10%.
  • Edge Performance: On the RDK S600 edge board, RTPolicy sustained 32.9 Hz (30.35 ms latency) in INT16 precision. On the entry-level RDK S100, it still managed a functional 6.7 Hz.
Comparison of end-to-end latency vs. mean success rate on edge devices. RTPolicy (the star) occupies the "sweet spot" of high accuracy and low latency.

Limitations and Future Work

While RTPolicy is a major step for edge robotics, the authors note a few areas for growth:

  • Task-Specific Training: Currently, the action model is trained on task-specific demonstrations. It is not yet a "generalist" policy that can handle any task out of the box.
  • Scaling to Generalist Control: The authors suggest that future work should involve pretraining the action model on massive, multi-robot datasets combined with language-conditioning signals to create a more universal "robot brain."
  • Data Requirements: Like most imitation learning methods, it still requires a significant number of human demonstrations (200 per task in the real world) to reach high performance.

Summary updated

Abstract

Low inference latency is critical for closed-loop robotic manipulation, especially under asynchronous inference, where action chunks must be updated promptly to correct accumulated deviations. However, existing visuomotor policies rely on computationally expensive visual backbones, resulting in high inference latency and deployment overhead. In this paper, we propose RTPolicy, a multi-view visuomotor policy that attains accurate manipulation with high inference efficiency. Its core design combines view-aware feature fusion with visual compression to aggregate complementary information across camera views into a compact token representation. The representation conditions a flow-matching action generator with the robot state and flow time, reducing the cost of action generation. Extensive experiments demonstrate that this design reduces the computational cost of visual representation and action generation while preserving visual information required for manipulation. On the RoboTwin~2.0 single-task clean evaluation across 50 tasks, RTPolicy attains a mean success rate of 52.26\%, outperforming the strongest leaderboard baseline $π_0$ (46.4\%). We further evaluate RTPolicy under asynchronous streaming inference on a real bimanual platform, where RTPolicy attains a mean success rate of 72.0\% and sustains real-time inference at 30.35 ms (32.9 Hz) on the RDK S600 and 149.72 ms (6.7 Hz) on the RDK S100, both in INT16. Project Page: https://d-robotics-ai-lab.github.io/RTPolicy.project.
BibTeX
@misc{hou2026rtpolicy,
  author = {Xianbao Hou and Zehao Ni and Yonghao He and Lingfeng Qian and Xin Shi and Hu Su and Wei Sui and Cong Yang},
  title = {{RTPolicy: Real-Time (32.9 Hz) Multi-View Visuomotor Policy for Robotic Manipulation on Edge Devices}},
  year = {2026},
  eprint = {2610.41625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41625},
}
Figure