TL;DR
HandUMI is an open-source, low-cost ($557/pair) hand-worn interface designed for "robot-free" bimanual data collection. By combining VR controller tracking with precise encoder-based parallel-jaw measurements, it allows researchers to capture high-quality demonstrations that can be retargeted to various robot embodiments, achieving a 60% success rate on real-world bimanual towel folding.
What Problem Does This Paper Address?
Scaling up robot learning requires massive amounts of demonstration data. Traditionally, this is done via teleoperation, which requires physical access to the robot for every single demonstration—a major bottleneck.
"Robot-free" interfaces like UMI have emerged to decouple data capture from the robot itself, but they often suffer from a "geometry mismatch." For example, if a human captures data using a scissor-like pinching motion but the robot uses translating parallel jaws, the resulting data might not transfer well. Furthermore, many existing solutions are either expensive, unimanual, or rely on less precise vision-based aperture estimation. HandUMI aims to provide a modular, bimanual, and highly accurate interface that matches the contact geometry of common research robots.
Methods
The HandUMI system consists of a 3D-printed wearable chassis that integrates several key components:
- Tracking: It leverages off-the-shelf VR controllers (Meta Quest 3S or PICO 4 Ultra) for robust 6-DoF hand tracking.
- Parallel-Jaw Interface: Unlike many "pincher" style interfaces, HandUMI uses a crank mechanism to drive translating parallel jaws, mimicking the motion of standard industrial grippers.
- Precise Sensing: A Feetech STS3215 12-bit servo encoder measures the jaw aperture directly, providing much higher fidelity than video-based estimation.
- Modularity: The fingertips are swappable, allowing the user to match the specific geometry of the target robot (e.g., AgileX PiPER, ARX X5).
- Retargeting Pipeline: The authors developed a pipeline using Inverse Kinematics (IK) to map the recorded hand trajectories onto seven different simulated bimanual robot embodiments.
Datasets
The authors collected a new dataset for a bimanual towel-folding task:
- HandUMI Towel Dataset: 562 successful robot-free demonstrations (approx. 5 hours of data) collected by three operators.
- Simulated Embodiments: The data was retargeted to seven different robot models in simulation (PiPER, OpenArm v1, TRLC-DK1, I2RT YAM, MakerMods Metal, Galaxea R1 Lite, and ARX X5) to test the versatility of the capture system.
Key Insights and Ideas
- Geometry Fidelity Matters: By matching the robot's parallel-jaw translation and specific fingertip shape, the authors reduce the sim-to-real gap inherent in robot-free data collection.
- Encoder vs. Vision: Using a dedicated encoder for jaw width provides reliable, low-latency data that is often missing from purely vision-based systems like UMI.
- Robot-Agnostic Capture: Because the recordings are stored as controller poses and jaw widths, a single demonstration session can be used to train multiple different robot types through the IK retargeting pipeline.
- Low Barrier to Entry: By using consumer VR hardware and 3D-printed parts, the system is significantly more accessible than professional teleoperation rigs.
Key Results
The system was validated on a real-world bimanual towel-folding task using AgileX PiPER arms:
- Success Rates: Policies trained using Action Chunking with Transformers (ACT) on the full 562-episode dataset achieved a 60% success rate (12/20) in unassisted rollouts.
- Fine-tuning: A fine-tuned $\pi_{0.5}$ model achieved a 35% success rate (7/20).
- Scaling Effect: Models trained on only 101 episodes failed completely (0/20), demonstrating that the ease of data collection provided by HandUMI is critical for reaching the data regime where these policies begin to work.
- Retargeting Precision: In simulation, the retargeting pipeline achieved a median position error of $\leq 1.2$ mm across all seven robot embodiments.
Limitations and Future Work
- Tracking Volume: The system is currently "headset-bounded," meaning the user must stay within the tracking volume of the VR headset.
- Gripper Variety: While modular, the current design is optimized for parallel-jaw grippers and does not yet support multi-fingered hands or complex tool use.
- Missing Modalities: The current version does not measure fingertip load (force sensing) or utilize audio data for training, though the hardware could potentially be extended to include these.
- Sim-to-Real Evidence: While retargeting was shown for seven robots in simulation, real-world validation was only performed on the PiPER arms. Future work could involve testing the transfer to the other six embodiments.
Summary updated