TL;DR
RTPolicy is a high-efficiency visuomotor policy that brings complex, multi-view robotic manipulation to edge devices. By combining view-aware feature fusion with aggressive visual compression and a flow-matching action generator, it achieves state-of-the-art accuracy while running at real-time speeds (up to 32.9 Hz) on low-power hardware.
What Problem Does This Paper Address?
For a robot to perform precise tasks—like inserting a component into a slot or stacking blocks—it needs to "see" from multiple angles and react instantly to what it sees. This is known as closed-loop, multi-view manipulation.
However, there is a major bottleneck: latency. Current high-performing policies use massive visual encoders and process hundreds of "tokens" (data chunks representing parts of an image). On powerful GPUs, this is fine, but on edge devices (the small computers actually mounted on robots), it’s too slow. If a policy takes 500ms to decide on an action, the robot is essentially "driving blind" for half a second, leading to accumulated errors and failed tasks. The authors aim to bridge this gap, providing a policy that is both accurate and fast enough for real-time deployment on edge boards.
Methods
RTPolicy introduces a streamlined pipeline designed to minimize the computational load at every step:
- Lightweight Encoding: It uses a frozen DINOv3 ConvNeXt encoder. Keeping the encoder frozen saves training time, while the ConvNeXt architecture is more efficient for edge deployment than standard Transformers.
- View-Aware Feature Fusion: Instead of processing each camera view in isolation, RTPolicy adds learnable "view embeddings" to the features. It then uses a Transformer block to let tokens from different cameras "talk" to each other. This ensures the model understands the 3D relationship between views before any data is discarded.
- Visual Compression: This is the "efficiency engine." After fusion, the model uses residual downsampling and adaptive pooling to shrink the visual representation. On their real-world platform, they reduce 900 dense tokens down to just 12 compact tokens—a 75-fold reduction in the data the action generator has to process.
- Flow-Matching Action Generator: Instead of a traditional Diffusion Policy that might require 10-50 iterations to "denoise" an action, RTPolicy uses a Diffusion Transformer (DiT) with a flow-matching objective. This allows it to generate high-quality action chunks in just 4 integration steps, drastically cutting inference time.
Datasets
The authors evaluated RTPolicy across both simulated and physical environments:
- RoboTwin 2.0: A large-scale simulation benchmark for bimanual (two-armed) manipulation. They tested on the "clean2clean" single-task leaderboard across 50 diverse tasks.
- Real-World Bimanual Platform: A custom setup using two ROKAE AR5 arms and three RealSense cameras (one head-mounted, two wrist-mounted). They designed three challenging tasks: Drag the Drawer, Stack and Cover, and Workpiece Assembly.
Key Insights and Ideas
- Token Count is the Latency Killer: The paper demonstrates that the number of visual tokens entering the action generator is a primary driver of latency. By compressing these tokens after cross-view fusion, they maintain spatial awareness while slashing compute costs.
- The "Streaming" Advantage: Low latency isn't just about speed; it's about accuracy. In their real-world tests, they found that running at 10 Hz (100ms updates) was significantly more successful than 2 Hz. Faster updates allow the robot to correct small deviations before they turn into task-ending failures.
- Fusion Before Compression: A critical design choice was performing feature fusion before compression. This allows the model to retain complementary information from different views that might otherwise be lost if each view were compressed independently.
Key Results
RTPolicy consistently outperformed existing baselines in both speed and success rate:
- Simulation: On the RoboTwin 2.0 benchmark, RTPolicy achieved a 52.26% mean success rate, surpassing the strongest baseline, $\pi_0$ (46.4%), and significantly beating Diffusion Policy (28.0%).
- Real-World: In bimanual tasks, it reached a 72.0% mean success rate. For the most difficult task (Workpiece Assembly), it achieved 30% success, while most baselines failed completely or stayed near 10%.
- Edge Performance: On the RDK S600 edge board, RTPolicy sustained 32.9 Hz (30.35 ms latency) in INT16 precision. On the entry-level RDK S100, it still managed a functional 6.7 Hz.
Limitations and Future Work
While RTPolicy is a major step for edge robotics, the authors note a few areas for growth:
- Task-Specific Training: Currently, the action model is trained on task-specific demonstrations. It is not yet a "generalist" policy that can handle any task out of the box.
- Scaling to Generalist Control: The authors suggest that future work should involve pretraining the action model on massive, multi-robot datasets combined with language-conditioning signals to create a more universal "robot brain."
- Data Requirements: Like most imitation learning methods, it still requires a significant number of human demonstrations (200 per task in the real world) to reach high performance.
Summary updated