Abstract
Modern robotic policies often struggle with semantic diversity and the high cost of human correction. We present RT-H (Robot Transformer with Action Hierarchies), a framework that decomposes high-level tasks into intermediate language-based motion descriptions before mapping to low-level control. This hierarchy allows for effective data sharing across tasks and enables a novel "language-based intervention" mechanism. Experimental results show that RT-H outperforms state-of-the-art flat models by 15% and significantly improves learning efficiency from human feedback.
Introduction
Current robotic learning typically employs "flat" architectures that map high-level commands (e.g., "pick up the coke can") and visual inputs directly to motor actions. While effective for simple, repetitive tasks, these models struggle as semantic diversity increases. A core challenge is the lack of explicit data sharing: while "Pick up a can" and "Pour a cup" are linguistically distinct, they share fundamental physical sub-movements. Flat models fail to capture this shared structure. Furthermore, correcting failures in these models usually requires expensive teleoperation, which is difficult to scale. RT-H addresses these issues by introducing a linguistic bottleneck that captures the shared "grammar" of physical movement.
Methodology
RT-H utilizes a Vision-Language Model (VLM) backbone, specifically PaLI-X 55B, to implement a two-stage hierarchical policy. Instead of a direct mapping, the process is split into two queries:
- Language Motion Query ($\pi_h$): Given the visual scene and high-level task, the model predicts a fine-grained language motion (e.g., "rotate arm right" or "close gripper"). This phrase describes the immediate physical intent.
- Action Query ($\pi_l$): This predicted motion, alongside the original task and image, is used to generate the actual 7D or 8D robot control tokens (positions, rotations, and gripper states).
To scale this approach without massive human labeling costs, the authors developed an automated pipeline that uses robot proprioception to "back-translate" motor movements into a library of over 2,500 language motion descriptions. To maintain real-time responsiveness, RT-H employs asynchronous querying, predicting the next motion while executing the current action.
Experimental Results
RT-H was evaluated on a massive multi-task dataset totaling 100,000 demonstrations, including a "Kitchen Dataset" and a new "Diverse Dataset" containing 24 complex semantic categories.
- Performance and Robustness: RT-H achieved a 15% average performance boost over the state-of-the-art flat model, RT-2. In high-precision tasks like "open pistachio jar" or "pull napkin from dispenser," RT-H maintained non-zero success rates where RT-2 often failed entirely.
- Learning from Intervention: A key innovation of RT-H is Language-Based Intervention. Because the model uses language as an intermediate step, humans can provide corrections via text (e.g., "move arm further left"). Fine-tuning on these corrections improved success rates from 40% to 63%, outperforming standard interactive imitation learning (IWR) by 50%.
- Generalization: The hierarchical approach allowed the robot to handle novel objects (e.g., OREO packages) with a 65% success rate, compared to 55% for RT-2.
Conclusion
By introducing a language-based action hierarchy, RT-H enables robots to learn shared physical structures across diverse tasks. This approach not only improves raw performance and generalization but also democratizes robot teaching by replacing complex teleoperation with simple linguistic feedback. While the current model relies on spatial language, future work will explore object-referential language to further enhance the intuitiveness of human-robot collaboration.
Summary updated