Deep Sensorimotor Control by Imitating Predictive Models of Human Motion

Himanshu Gaurav Singh, Pieter Abbeel, Jitendra Malik, Antonio LoquercioarXiv 2025 · DOI · arXiv

Abstract

Training dexterous robot manipulation policies often requires labor-intensive reward engineering or complex kinematic retargeting of human demonstrations. This paper introduces a scalable alternative: using a predictive model of human motion to guide robot learning. By representing both humans and robots through a shared abstraction of 3D fingertip keypoints, a motion predictor trained on human datasets can be used zero-shot to provide dense tracking rewards. This approach enables robots to learn complex tasks like grasping, lifting, and opening cabinets without manual reward design, demonstrating successful cross-embodiment transfer across various robotic hands.

Introduction

The challenge of teaching robots dexterous manipulation lies in the "reward gap." Sparse rewards (e.g., "did the object move?") are easy to define but difficult to learn from, while dense rewards (step-by-step instructions) require significant engineering effort. While large-scale human video datasets exist, mapping human motion to robot joints (retargeting) is technically demanding and environment-specific. This work proposes bypassing these hurdles by focusing on fingertip trajectories—a "universal language" shared by humans and anthropomorphic robots.

Methodology: Human Motion Prediction

The core of the framework is the Human Motion Predictor ($\Pi_h$). This model is a causal transformer trained on the DexYCB dataset, which contains 1,000 trajectories of human hand-object interactions.

Architecture of the Human Motion Predictor ($\Pi_h$) trained on human interaction datasets.

The predictor takes a history of 3D fingertip locations ($k^h$) and object point clouds ($o$) to forecast future fingertip positions ($\widehat{k}^h$). Because the model operates on 3D coordinates rather than joint angles, it can be applied directly to robotic hands that share a similar fingertip configuration, regardless of their internal degrees of freedom (DoF).

Reward Formulation and Policy Optimization

During reinforcement learning (RL) in a simulated environment (IsaacGym), the robot's policy $\pi$ is optimized using Proximal Policy Optimization (PPO). The learning process is guided by a composite reward function:

  1. Tracking Reward ($r_{track}$): The robot receives a reward for minimizing the Euclidean distance between its current fingertip positions ($k^r$) and the "human-like" positions predicted by the frozen $\Pi_h$.
  2. Task Reward ($r_{task}$): A sparse signal indicating task completion.
Reinforcement learning framework integrating the frozen motion predictor to generate dense tracking rewards.

This setup allows the motion predictor to act as a "universal coach," providing a natural curriculum. For instance, in a drawer-opening task, the predictor naturally guides the hand toward the handle before encouraging a pulling motion, as observed in human data.

Experimental Results

The proposed method was evaluated across several challenging tasks, including Grasp and Lift, Lift and Throw, and Open Cabinet. Key findings include:

  • Cross-Embodiment Transfer: The same human-trained predictor successfully guided three different robot hands: the Allegro (16-DoF), Xhand (12-DoF), and SVH (20-DoF).
  • Performance: Success rates were comparable to "privileged" baselines that used manually tuned dense rewards.
  • Generalization: The model provided useful guidance even for tasks not explicitly present in the training data, such as throwing objects.

Discussion and Limitations

While effective, the framework currently requires the robot to have an anthropomorphic form factor to utilize the fingertip keypoint abstraction. Future work is needed to extend this to non-humanoid grippers. Additionally, while the policies show promise in simulation, real-world deployment (sim-to-real) remains a challenge due to potential sensor noise and the requirement for accurate object point cloud reconstruction.

Conclusion

By leveraging human motion prediction as a dense reward signal, this work offers a path toward more scalable robot learning. Decoupling the human data from the RL training loop allows for a "train once, use anywhere" model that significantly reduces the burden of reward engineering for dexterous manipulation.

Summary updated

Abstract

As the embodiment gap between a robot and a human narrows, new opportunities arise to leverage datasets of humans interacting with their surroundings for robot learning. We propose a novel technique for training sensorimotor policies with reinforcement learning by imitating predictive models of human motions. Our key insight is that the motion of keypoints on human-inspired robot end-effectors closely mirrors the motion of corresponding human body keypoints. This enables us to use a model trained to predict future motion on human data \emph{zero-shot} on robot data. We train sensorimotor policies to track the predictions of such a model, conditioned on a history of past robot states, while optimizing a relatively sparse task reward. This approach entirely bypasses gradient-based kinematic retargeting and adversarial losses, which limit existing methods from fully leveraging the scale and diversity of modern human-scene interaction datasets. Empirically, we find that our approach can work across robots and tasks, outperforming existing baselines by a large margin. In addition, we find that tracking a human motion model can substitute for carefully designed dense rewards and curricula in manipulation tasks. Code, data and qualitative results available at https://jirl-upenn.github.io/track_reward/.
BibTeX
@misc{singh2025deep,
  author = {Himanshu Gaurav Singh and Pieter Abbeel and Jitendra Malik and Antonio Loquercio},
  title = {{Deep Sensorimotor Control by Imitating Predictive Models of Human Motion}},
  year = {2025},
  eprint = {2508.18691},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.18691},
}
Figure