Haptic-ACT: Bridging Human Intuition with Compliant Robotic Manipulation via Immersive VR

Kelin Li, Shubham M. Wagh, Nitish Sharma, Saksham Bhadani, Wei Chen, Chang Liu, Petar Kormushev

Abstract

Imitation learning in robotics is often hampered by the quality of teleoperated demonstrations, which typically lack haptic feedback. This absence of touch leads human operators to apply excessive force, resulting in "aggressive" data that prevents robots from learning delicate, compliant manipulation. We present Haptic-ACT, a framework that integrates immersive VR and haptic gloves into the data collection pipeline. By incorporating fingertip force data into a transformer-based imitation learning model, we enable robots to perform tasks with significantly reduced grasping forces (15% reduction) while maintaining high success rates (86.2%).

Introduction

The performance of robotic imitation learning models is fundamentally tied to the quality of human demonstrations. While teleoperation is a common method for data collection, most existing systems provide only visual feedback. Without haptic cues, demonstrators cannot perceive contact forces, leading them to over-compensate with excessive pressure. This results in training data that lacks the nuance required for handling fragile or deformable objects like fruit or paper cups.

Immersive VR Teleoperation System

To solve the feedback gap, we developed a teleoperation system using a Meta Quest 3 headset for visual immersion and a SenseGlove Nova for haptic feedback. This setup allows the demonstrator to see the environment in 3D while receiving tactile cues from the robot's interactions.

The Haptic-ACT teleoperation system, showing the real-world robot setup (a), the human demonstrator with VR and haptic gear (b), and the corresponding immersive VR view (c).

The system employs a "digital twin" approach to manage control latency and uses a third-order polynomial regression to map robot motor values to fingertip forces. These forces are then translated into physical resistance in the operator's glove, allowing them to "feel" the object being manipulated in real-time.

The Haptic-ACT Framework

Haptic-ACT extends the popular Action Chunking with Transformers (ACT) architecture by integrating force as a primary input feature. The model processes multi-modal inputs—including images, joint positions, and a $5 \times 1$ fingertip force vector—through a Conditional Variational Autoencoder (CVAE) and a Transformer-based encoder-decoder. This allows the robot to learn not just the trajectory of a movement, but the intended "compliance" or softness of the contact, mimicking human intent rather than just physical paths.

Experimental Results

We evaluated the system on a pick-and-place task in both simulation and real-world environments. Data was collected from five individuals with varying levels of expertise to ensure a robust dataset.

Fingertip force profiles over time. (a) Comparison between Haptic-ACT, standard ACT, and human demonstrations. (b) Impact of haptic feedback (SenseGlove) on human demonstration force.

Key results include:

  • Demonstration Quality: Human operators applied 25% less force when provided with haptic feedback, producing more consistent and "gentle" data.
  • Model Performance: Haptic-ACT reduced average grasping forces by approximately 15% compared to the original ACT model.
  • Success Rates: The model achieved an 86.2% success rate in real-world tasks, outperforming the non-haptic baseline (83.3%).
  • Digit Specificity: Statistical analysis confirmed that force reduction was most significant in the thumb and index fingers, the primary digits used for stable, delicate grasping.

Conclusion and Future Work

Haptic-ACT demonstrates that haptic feedback is a critical component for high-quality robotic imitation learning. By allowing humans to use their natural intuition for force control, we can train robots to be more compliant and precise. Future work will explore more complex, contact-rich tasks such as drilling or pouring, and the integration of 3D vision models to enhance generalization in unstructured environments.

Summary updated

BibTeX
@inproceedings{li2025haptic,
  author = {Kelin Li and Shubham M. Wagh and Nitish Sharma and Saksham Bhadani and Wei Chen and Chang Liu and Petar Kormushev},
  title = {{Haptic-ACT: Bridging Human Intuition with Compliant Robotic Manipulation via Immersive VR}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11247471},
}
Figure