Correcting Persistent Failures in Point-Cloud Grasping Policies using Human Demonstrations in Virtual Reality

Achyuthan Unni Krishnan, Karthik Rengarajan, Bharath Narasimhan, Chris von CsefalvayarXiv 2026 · arXiv

TL;DR

Researchers have developed a "failure loop" that automatically identifies specific object poses where reinforcement learning (RL) policies persistently fail. By using a VR interface, humans provide just two contact points for these hard cases, allowing the robot to nearly double its success rate on difficult grasps without sacrificing its overall performance on new objects.

What Problem Does This Paper Address?

Modern robots often use point-cloud-based grasping policies to pick up unfamiliar objects. These systems typically start with an offline "grasp generator" (like Contact-GraspNet) that suggests plausible hand positions based on geometry. While Reinforcement Learning (RL) can significantly improve these policies by practicing in simulation, it suffers from a frustrating trade-off: as the robot gets better on average, it often "forgets" how to solve specific, difficult cases it could previously handle.

The core challenge is two-fold:

  1. Efficiency: Human time is expensive. We shouldn't waste it demonstrating grasps the robot can already do.
  2. Kinematic Mismatch: A human hand moves differently than a parallel-jaw robot gripper. Directly copying a human's arm motion is often counterproductive.

Methods

The authors propose a human-in-the-loop failure loop consisting of four main stages:

  1. Failure Mining: During RL training in the Isaac Lab simulation, the system tracks every failure. If a specific object pose fails four times in a row, it is flagged as a "hard case" and exported.
  2. VR Demonstration: A human wearing a Meta Quest 3 headset enters a virtual environment containing the failed case. The human picks up the object naturally. Crucially, the system only records the two contact points where the fingertips touched the object, ignoring the human's specific arm trajectory.
  3. Robot Check: Before the demonstration is used for training, an automated server tests if the robot's own gripper can actually succeed using those two points at various approach angles. This ensures the human's "advice" is physically compatible with the robot's hardware.
  4. Retraining with Imitation: The robot is retrained from its original starting point. It uses a hybrid loss function: standard RL (PPO) to explore, plus an imitation loss that encourages it to hit the human-provided contact points. This imitation "fades out" over time, allowing the RL to fine-tune the final policy.

Datasets

The study utilized:

  • YCB Object and Model Set: A standard robotics benchmark featuring real-world objects like mugs, tools, and food containers.
  • Procedural Shapes: Synthetic objects like brackets, tori, and star prisms to increase geometric variety.
  • Training/Test Split: Training involved 451 instances from 57 families, while evaluation was performed on 24 separate YCB families never seen during training.

Key Insights and Ideas

  • "Where, not How": By only providing contact points rather than full trajectories, the researchers bypassed the "correspondence problem." The human provides the intent (where to squeeze), and the robot uses its own motion planner to figure out the approach.
  • RL as a Double-Edged Sword: The paper highlights that RL alone, while raising average success from 55% to 83%, actually dropped success on "hard cases" to just 24%. This suggests that standard RL optimization can be "lazy," sacrificing difficult outliers to maximize the mean.
  • Generalization of Demonstrations: Interestingly, demonstrating a grasp for one specific pose helped the robot succeed on other poses of the same object that were never demonstrated. The human input provided a geometric "hint" that the neural network could generalize.

Key Results

  • Overall Performance: On unseen test objects, the RL-based policy improved success from 55% (baseline Contact-GraspNet) to 83%.
  • Hard Case Recovery: On the specific cases where RL previously failed, adding just 58 human demonstrations nearly doubled the success rate from 24% to 45%.
  • Zero Regression: The addition of these targeted demonstrations did not hurt the robot's performance on general, unseen objects, proving that the "failure loop" is an effective way to patch holes in a policy without breaking the rest of it.
  • Real-Arm Testing: While most training was done with a "floating hand," the authors validated the final policy on a simulated Franka Emika Research 3 arm, showing it could successfully sort objects into bins.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Simulation-to-Real Gap: All experiments were conducted in simulation. While the physics are high-fidelity, real-world sensor noise and contact dynamics may pose new challenges.
  • Observation Coverage: The current policy assumes a fairly complete point cloud. In real-world scenarios, a robot often only sees one side of an object, which might require fusing multiple camera views.
  • Kinematic Scope: The system currently targets objects that fit within a standard parallel-jaw gripper. Handling very large or highly complex objects would require different gripper models.
  • Automation: Future work could involve replacing the human in the loop with an automated "teacher" (like a slower, more computationally expensive grasp planner) to see if the same failure loop can be fully autonomous.

Summary updated

Abstract

Robots in homes, warehouses and farms must pick up objects they have never seen, usually without a 3D model or a category label. A grasping policy that works directly on the point cloud from a depth sensor needs only the geometry the robot observes, so it applies to any new object. Offline grasp generators such as Contact-GraspNet work this way, but they rank grasps by learned plausibility rather than by whether the grasp actually lifts the object. Reinforcement learning (RL) can optimize for that directly and make a point-cloud grasping policy much better on average, yet it leaves specific grasps persistently unsolved. We present a human-in-the-loop failure loop that targets those grasps: situations an RL policy keeps failing are mined automatically, sent to a virtual-reality application where a person picks the object up with their own hands, and returned to training as two grasp points. The person contributes only where to grasp; the robot chooses its own approach. In simulation, RL raises success on unseen objects from 55% with Contact-GraspNet to 83%. Adding 58 human demonstrations nearly doubles success on the hard cases RL kept failing, from 24% to 45%, and recovers performance that RL alone loses on similar cases that were never demonstrated, without reducing success on unseen objects.
BibTeX
@misc{krishnan2026correcting,
  author = {Achyuthan Unni Krishnan and Karthik Rengarajan and Bharath Narasimhan and Chris von Csefalvay},
  title = {{Correcting Persistent Failures in Point-Cloud Grasping Policies using Human Demonstrations in Virtual Reality}},
  year = {2026},
  eprint = {2611.18919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18919},
}
Figure