HandUMI: A Modular Hand-Worn Parallel-Jaw Interface for Robot-Free Bimanual Data Capture

Brik Meza, Leonardo Perez, Alvaro Mendoza-Li, Aniela Ubillus, Mitshell Ramos, Raul Bastidas, Mauricio Olivera, Bryan BastidasarXiv 2026 · arXiv

TL;DR

HandUMI is an open-source, low-cost ($557/pair) hand-worn interface designed for "robot-free" bimanual data collection. By combining VR controller tracking with precise encoder-based parallel-jaw measurements, it allows researchers to capture high-quality demonstrations that can be retargeted to various robot embodiments, achieving a 60% success rate on real-world bimanual towel folding.

What Problem Does This Paper Address?

Scaling up robot learning requires massive amounts of demonstration data. Traditionally, this is done via teleoperation, which requires physical access to the robot for every single demonstration—a major bottleneck.

"Robot-free" interfaces like UMI have emerged to decouple data capture from the robot itself, but they often suffer from a "geometry mismatch." For example, if a human captures data using a scissor-like pinching motion but the robot uses translating parallel jaws, the resulting data might not transfer well. Furthermore, many existing solutions are either expensive, unimanual, or rely on less precise vision-based aperture estimation. HandUMI aims to provide a modular, bimanual, and highly accurate interface that matches the contact geometry of common research robots.

Methods

The HandUMI system consists of a 3D-printed wearable chassis that integrates several key components:

  • Tracking: It leverages off-the-shelf VR controllers (Meta Quest 3S or PICO 4 Ultra) for robust 6-DoF hand tracking.
  • Parallel-Jaw Interface: Unlike many "pincher" style interfaces, HandUMI uses a crank mechanism to drive translating parallel jaws, mimicking the motion of standard industrial grippers.
  • Precise Sensing: A Feetech STS3215 12-bit servo encoder measures the jaw aperture directly, providing much higher fidelity than video-based estimation.
  • Modularity: The fingertips are swappable, allowing the user to match the specific geometry of the target robot (e.g., AgileX PiPER, ARX X5).
  • Retargeting Pipeline: The authors developed a pipeline using Inverse Kinematics (IK) to map the recorded hand trajectories onto seven different simulated bimanual robot embodiments.
Figure 1: HandUMI. Left: robot-free capture with the hand-worn device (fisheye camera, servo encoder, swappable fingertips) tracked by PICO 4 Ultra or Meta Quest controllers. Right: recorded trajectories retargeted in simulation to seven bimanual arms.

Datasets

The authors collected a new dataset for a bimanual towel-folding task:

  • HandUMI Towel Dataset: 562 successful robot-free demonstrations (approx. 5 hours of data) collected by three operators.
  • Simulated Embodiments: The data was retargeted to seven different robot models in simulation (PiPER, OpenArm v1, TRLC-DK1, I2RT YAM, MakerMods Metal, Galaxea R1 Lite, and ARX X5) to test the versatility of the capture system.

Key Insights and Ideas

  • Geometry Fidelity Matters: By matching the robot's parallel-jaw translation and specific fingertip shape, the authors reduce the sim-to-real gap inherent in robot-free data collection.
  • Encoder vs. Vision: Using a dedicated encoder for jaw width provides reliable, low-latency data that is often missing from purely vision-based systems like UMI.
  • Robot-Agnostic Capture: Because the recordings are stored as controller poses and jaw widths, a single demonstration session can be used to train multiple different robot types through the IK retargeting pipeline.
  • Low Barrier to Entry: By using consumer VR hardware and 3D-printed parts, the system is significantly more accessible than professional teleoperation rigs.

Key Results

The system was validated on a real-world bimanual towel-folding task using AgileX PiPER arms:

  • Success Rates: Policies trained using Action Chunking with Transformers (ACT) on the full 562-episode dataset achieved a 60% success rate (12/20) in unassisted rollouts.
  • Fine-tuning: A fine-tuned $\pi_{0.5}$ model achieved a 35% success rate (7/20).
  • Scaling Effect: Models trained on only 101 episodes failed completely (0/20), demonstrating that the ease of data collection provided by HandUMI is critical for reaching the data regime where these policies begin to work.
  • Retargeting Precision: In simulation, the retargeting pipeline achieved a median position error of $\leq 1.2$ mm across all seven robot embodiments.
Figure 2: Towel folding: views, a rollout, and successes by training episodes (Wilson 95% CI).

Limitations and Future Work

  • Tracking Volume: The system is currently "headset-bounded," meaning the user must stay within the tracking volume of the VR headset.
  • Gripper Variety: While modular, the current design is optimized for parallel-jaw grippers and does not yet support multi-fingered hands or complex tool use.
  • Missing Modalities: The current version does not measure fingertip load (force sensing) or utilize audio data for training, though the hardware could potentially be extended to include these.
  • Sim-to-Real Evidence: While retargeting was shown for seven robots in simulation, real-world validation was only performed on the PiPER arms. Future work could involve testing the transfer to the other six embodiments.

Summary updated

Abstract

HandUMI captures robot-free bimanual demonstrations with a hand-worn parallel-jaw interface. Swappable fingertips match the robot's installed gripper; Quest/PICO controllers track hands and encoders measure aperture. Robot-agnostic recordings are retargeted by inverse kinematics to seven simulated bimanual arms (median position error $\leq 1.2$ mm; $\leq 4.7$% of episodes rejected). On bimanual towel folding, 562 robot-free demonstrations train policies on real PiPER arms: ACT succeeds in 12/20 attempts and fine-tuned $π_{0.5}$ in 7/20; ACT on the first 101 and base $π_{0.5}$ each succeed in 0/20. Hardware and software are open source (Apache-2.0), and a pair costs USD 557 with Quest 3S, excluding the optional ZED Mini and host computer. Project website: https://murobotics.ai/research/handumi.
BibTeX
@misc{meza2026handumi,
  author = {Brik Meza and Leonardo Perez and Alvaro Mendoza-Li and Aniela Ubillus and Mitshell Ramos and Raul Bastidas and Mauricio Olivera and Bryan Bastidas},
  title = {{HandUMI: A Modular Hand-Worn Parallel-Jaw Interface for Robot-Free Bimanual Data Capture}},
  year = {2026},
  eprint = {2611.22146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22146},
}
Figure