TL;DR
UMI-Live is an open-source, self-contained hardware and software system designed to scale "in-the-wild" bimanual robot data collection. By integrating LiDAR-inertial sensing and onboard compute directly into handheld grippers, it enables researchers to collect high-quality demonstrations in any environment and automatically process them into training-ready datasets for state-of-the-art robot policies.
What Problem Does This Paper Address?
Modern robot learning, particularly imitation learning, is data-hungry. While vision-language-action (VLA) models show great promise, they require massive, diverse datasets of human demonstrations. Traditionally, this data is collected via teleoperation, which is cumbersome because it requires the physical robot, safety infrastructure, and a fixed setup to be present at every collection site.
The Universal Manipulation Interface (UMI) concept allows humans to demonstrate tasks using handheld grippers instead of robots. However, previous UMI iterations often relied on external tracking rigs, bulky backpack-mounted computers, or tedious manual data processing steps. UMI-Live aims to bridge this gap by providing a "one-button" recording experience that is truly portable and automates the path from demonstration to deployment.
Methods
The core of UMI-Live is a pair of custom-designed handheld grippers. Unlike previous versions that might rely on visual SLAM alone, each UMI-Live unit integrates:
- Sensing: A RoboSense Airy LiDAR with an integrated IMU and a global-shutter wide-angle USB camera.
- Compute: A Raspberry Pi 5 for onboard data logging and an LCD for status updates.
- Mechanism: A redesigned parallel-jaw gripper using ArUco tags on the finger mounts to visually track jaw opening.
The software side features an asynchronous processing pipeline. As soon as a recording is finished, it is transferred to a server that performs parallel LiDAR-inertial SLAM (using the GLIM framework) to reconstruct 6-DOF trajectories. This allows the operator to continue collecting data while the system automatically handles synchronization, pose estimation, and dataset formatting.
Datasets
The authors have released a substantial bimanual dataset to the community:
- Total Volume: ~40 hours of data consisting of 3,255 bimanual episodes.
- Tasks: The data covers orange placement, water pouring, and towel folding.
- Format: Data is published in the LeRobot format on Hugging Face, ensuring compatibility with modern robotics training pipelines.
For their physical evaluation, they used a mixture of 296 UMI demonstrations and 75 robot teleoperation demonstrations for a specific "orange-to-bowl" task.
Key Insights and Ideas
- Self-Containment Scales: By putting the battery, compute, and sensing on the gripper, the system removes the "tether" to a lab. This allows for data collection in diverse, real-world lighting and geometric conditions.
- LiDAR-Inertial Robustness: Moving beyond pure vision-based tracking to LiDAR-inertial SLAM provides more reliable 6-DOF trajectories, especially in environments where visual features might be sparse or lighting is inconsistent.
- Asynchronous Throughput: The decoupling of collection and processing is a major UX improvement. An operator can record dozens of episodes in a row without waiting for the heavy compute of SLAM reconstruction to finish for each one.
- Policy Fine-tuning: The paper demonstrates that VLA models like $\pi_{0.5}$ and MolmoAct2 can be effectively fine-tuned on UMI data collected in one environment and successfully deployed on a robot in a different environment, provided a small amount of target-domain teleoperation data is included in the mix.
Key Results
The system was validated through physical robot rollouts on dual i2RT YAM arms performing a bimanual pick-and-place task (moving six oranges into a bowl).
- $\pi_{0.5}$ Performance: Achieved a 10/10 success rate when trained on a 90% UMI / 10% teleop data split. Even with only 5% teleop data, it maintained a 9/10 success rate.
- MolmoAct2 Performance: Achieved an 8/10 success rate after fine-tuning, compared to 0/10 for the base released checkpoint, proving the value of the UMI-Live data for adapting large models to new tasks.
- Hardware Cost: The system is relatively affordable for research labs, costing approximately $1,930 per pair of grippers.
Limitations and Future Work
While UMI-Live is a significant step forward, the authors note several areas for improvement:
- Ergonomics: Integrating all components onto the handle increases the mass to 1.42 kg per unit, which can lead to operator fatigue during long sessions.
- Calibration: The devices still require a one-time manual calibration for the camera and gripper.
- SLAM Dependencies: The quality of the trajectory reconstruction is still dependent on the scene having enough geometric features for the LiDAR to track.
- Evaluation Breadth: Physical evaluation was limited to one task and one training seed per condition; broader testing across more diverse tasks is needed to fully characterize the system's limits.
Summary updated