Teaching Robots to Grasp Like Humans: An Interactive Approach

Anna Mészáros, Giovanni Franzese, Jens Kober

Abstract

This work investigates how the intricate task of a continuous pick & place (P&P) motion may be learned from humans based on demonstrations and corrections. Due to the complexity of the task, these demonstrations are often slow and even slightly flawed, particularly at moments when multiple aspects (i.e., end-effector movement, orientation, and gripper width) have to be demonstrated at once. Rather than training a person to give better demonstrations, non-expert users are provided with the ability to interactively modify the dynamics of their initial demonstration through teleoperated corrective feedback. This in turn allows them to teach motions outside of their own physical capabilities. In the end, the goal is to obtain a faster but reliable execution of the task. The presented framework learns the desired movement dynamics based on the current Cartesian position with Gaussian Processes (GPs), resulting in a reactive, time-invariant policy. Using GPs also allows online interactive corrections and active disturbance rejection through epistemic uncertainty minimization. The experimental evaluation of the framework is carried out on a Franka-Emika Panda. Tests were performed to determine i) the framework's effectiveness in successfully learning how to quickly pick & place an object, ii) ease of policy correction to environmental changes (i.e., different object sizes and mass), and iii) the framework's usability for non-expert users.

Introduction: Overcoming the "Stop-and-Go" Paradigm

In traditional industrial robotics, pick-and-place (P&P) tasks are often characterized by a "stop-and-go" strategy: the robot approaches an object, stops completely to engage the gripper, and then accelerates toward the destination. While safe, this approach is highly inefficient compared to the fluid, continuous motions humans naturally employ. Bridging this gap is challenging because kinesthetic teaching—physically guiding a robot arm—is inherently slow and awkward. A human teacher cannot easily demonstrate high-speed, high-precision grasps while fighting the robot's mechanical resistance. Furthermore, simply speeding up a slow demonstration fails due to impact forces and communication latencies that cause the gripper to miss.

This paper proposes a framework where a non-expert user provides a "rough draft" of a motion and then "sculpts" it into a high-speed, reliable policy through interactive joystick feedback.

Methodology: Interactive Dynamics Sculpting

The core of the framework is a reactive, time-invariant policy learned via Gaussian Process (GP) dynamics. Instead of following a fixed sequence of timestamps, the robot learns to determine its next move based on its current Cartesian position ($x$).

The proposed control framework integrating Gaussian Process dynamics and interactive human corrections.

Key components include:

  • Minimum Uncertainty Dynamical System (MUDS): The robot calculates "epistemic uncertainty"—a measure of how much it lacks data in its current region. A "stabilization field" pulls the robot back toward high-confidence regions, preventing it from drifting into dangerous, unknown states during high-speed execution.
  • Decoupled Interactive Correction: After an initial demonstration, the user observes the robot's attempts and provides corrections via a gamepad. Crucially, the user can adjust the path shape and the speed scaling factor ($\gamma$) independently, allowing for intuitive refinement.
  • Minimum Uncertainty Inference: For gripper width and orientation, the system projects the robot's state onto the most similar known state in its database, ensuring consistent behavior even in slightly novel positions.

Key Insights: Uncertainty as a Behavioral Guide

A striking result of this approach is that humans can teach robots to perform tasks better than the humans themselves can demonstrate. By providing corrections to an active policy, users can push the robot to speeds that would be physically impossible to demonstrate kinesthetically.

Comparison of the original demonstration and the trained policy, showing the stabilization field and executed trajectory.

The authors treat the gradient of uncertainty as a potential field, creating a form of "behavioral stiffness." This keeps the robot on track during high-speed maneuvers or external perturbations without requiring the rigid, high-gain controllers that typically make robots hazardous to work near. Decoupling path shape from speed allows non-experts to focus on one aspect of the skill at a time, mimicking human skill acquisition.

Experimental Evaluation and Results

The framework was tested on a Franka-Emika Panda robot using a variety of objects, including a 250g rigid bottle, a heavy 900g bottle, and various flexible or deformable items. Ten participants (ages 23–28) were recruited for a user study to evaluate the system's accessibility.

Key Results: * Significant Speedup: Execution times were reduced from an 11-second human demonstration to under 3 seconds for autonomous execution—a nearly 4x improvement. * Reliability: The system achieved an 82% average success rate across 50 autonomous trials. * Ablation Study: The MUDS feature proved critical; without it, success rates plummeted from 95% to 45%, and training time tripled. * Generalization: The system adapted to a 900g object with zero additional corrections and handled deformable objects significantly faster than learning from scratch.

Limitations and Future Work

While effective, the current gamepad interface limits the user's ability to correct gripper orientation post-demonstration. Future work aims to enable orientation corrections and integrate obstacle avoidance into the vector field approach. Additionally, adding haptic feedback to the teacher could further enhance the quality of interactive corrections.

Summary updated

Abstract

This work investigates how the intricate task of a continuous pick & place (P&P) motion may be learned from humans based on demonstrations and corrections. Due to the complexity of the task, these demonstrations are often slow and even slightly flawed, particularly at moments when multiple aspects (i.e., end-effector movement, orientation, and gripper width) have to be demonstrated at once. Rather than training a person to give better demonstrations, non-expert users are provided with the ability to interactively modify the dynamics of their initial demonstration through teleoperated corrective feedback. This in turn allows them to teach motions outside of their own physical capabilities. In the end, the goal is to obtain a faster but reliable execution of the task. The presented framework learns the desired movement dynamics based on the current Cartesian position with Gaussian Processes (GPs), resulting in a reactive, time-invariant policy. Using GPs also allows online interactive corrections and active disturbance rejection through epistemic uncertainty minimization. The experimental evaluation of the framework is carried out on a Franka-Emika Panda. Tests were performed to determine i) the framework's effectiveness in successfully learning how to quickly pick & place an object, ii) ease of policy correction to environmental changes (i.e., different object sizes and mass), and iii) the framework's usability for non-expert users.
BibTeX
@article{meszaros2022teaching,
  author = {Anna Mészáros and Giovanni Franzese and Jens Kober},
  title = {{Teaching Robots to Grasp Like Humans: An Interactive Approach}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2022.3165531},
}
Figure