Anticipating Human Intention for Full-Body Motion Prediction in Object Grasping and Placing Tasks

Philipp Kratzer, Niteesh Balachandra Midlagajni, Marc Toussaint, Jim Mainprice

Introduction

For robots to work safely and efficiently alongside humans, they must be able to "see into the future" to avoid collisions and coordinate tasks. While short-term motion prediction (the next few hundred milliseconds) is relatively well-handled by standard recurrent neural networks (RNNs), long-term prediction remains a significant challenge. Human movement is not merely a sequence of joint angles; it is driven by intent and constrained by the environment. For instance, a person reaching for a cup has a trajectory determined by the cup's location and available clear space on a table. Most existing models ignore this environmental context, leading to "uninformed" predictions that drift into obstacles or fail to reach the intended goal.

This research, presented at ICRA 2020, proposes a framework that predicts human motion by first anticipating "intent" through environmental affordances—the possible actions a scene allows. By combining deep learning (RNNs and CNNs) with trajectory optimization, the system forecasts complex, full-body manipulation movements that respect the 3D geometry of the environment.

Methods: The Offline-Online Pipeline

The researchers developed a two-stage pipeline consisting of an offline training phase and an online prediction phase. The offline phase involves training affordance models (for graspability and placeability) and a full-body motion prediction RNN using a large motion database.

The online prediction pipeline combining affordance modeling, trajectory prediction, and optimization.

Affordance Modeling

The system identifies "where" an action might occur:

  • Placeability: A Mixture Density Network (MDN) combined with a CNN processes 2D occupancy maps and Signed Distance Fields (SDF) of surfaces to identify valid placement zones. The MDN allows the model to represent multiple possible goals (multi-modality), acknowledging that a person could place an object in several different spots.
  • Graspability: The likelihood of wrist positions around an object is modeled using both Gaussian and von Mises-Fisher (vMF) distributions, the latter being particularly suited for modeling points on spherical surfaces.

Motion Prediction and Optimization

The system then determines "how" the motion is performed:

  • Trajectory Generation: A VRED (Position-Velocity Recurrent Encoder-Decoder) network generates a base kinematic trajectory.
  • Trajectory Optimization: At runtime, the system samples a goal from the affordance model and uses a non-linear optimizer (L-BFGS) to refine the VRED's predicted trajectory. This ensures the final motion is kinematically natural while strictly satisfying the "goal constraint" (e.g., the hand actually ends up at the predicted grasp point).

Experimental Setup

The authors collected a custom dataset using an Optitrack motion capture system in a $4 \times 4$ meter area. Five subjects wearing 50-marker MoCap suits performed 1,551 grasp-place sequences, such as setting and clearing a table with cups, jugs, plates, and bowls. The environment included a table, a large shelf, and a small shelf to provide varied heights and surfaces.

Key Insights

  • Geometry as a First-Class Citizen: By incorporating Signed Distance Fields (SDFs) into the affordance model, the system learns to avoid placing objects where they would overlap with existing ones.
  • The Power of Multi-modality: Using MDNs is crucial. A standard regressor might predict the "average" of all valid spots, which often lands in the middle of an obstacle. MDNs maintain distinct "peaks" of probability for different valid regions.
  • Transfer Learning for Spatial Awareness: Pre-training an autoencoder to reconstruct occupancy maps helped the "Placeability" network learn more robust features about the environment's layout, leading to a 5% improvement in predicting valid placement regions.
  • Optimization as a Bridge: Purely data-driven RNNs often "miss" the goal slightly. Using the RNN as a prior for a trajectory optimizer provides the best of both worlds: natural-looking motion that actually completes the task.

Results

The proposed method significantly outperformed baselines in both affordance accuracy and full-body motion prediction.

Comparison of predicted human motion sequences for grasping and placing tasks.
  • Affordance Accuracy: The "Transfer Learning + Penalty" model for placeability significantly outperformed baselines, especially when the human was still far from the table (4 seconds before placement).
  • Grasping: Both neural models (Gaussian and vMF) beat the heuristic baseline by a wide margin. While the Gaussian model had slightly lower mean squared error, the vMF model provided more useful directional uncertainty for the approach angle.
  • Full-Body Prediction: When predicting 1.5 seconds into the future, the proposed method achieved an error rate nearly identical to an "oracle" (a model given the true ground-truth goal). It significantly outperformed a standard VRED RNN, which tended to "undershoot" the movement.

Limitations and Future Work

Current limitations include the small participant pool (5 subjects) and the focus on immediate next-action prediction. Future work could integrate higher-level symbolic planning to predict sequences of actions (e.g., "pick up the bottle, then pour into the glass"). Additionally, transitioning the system to work with standard RGB-D cameras in real-time is a necessary step for practical robotics.

Summary updated

Abstract

Motion prediction in unstructured environments is a difficult problem and is essential for safe and efficient human-robot space sharing and collaboration. In this work, we focus on manipulation movements in environments such as homes, workplaces or restaurants, where the overall task and environment can be leveraged to produce accurate motion prediction. For these cases we propose an algorithmic framework that accounts explicitly for the environment geometry based on a model of affordances and a model of short-term human dynamics both trained on motion capture data. We propose dedicated function networks for graspability and placebility affordances and we make use of a dedicated RNN [1] for short-term motion prediction. The prediction of grasp and placement probability densities are used by a constraint-based trajectory optimizer to produce a full-body motion prediction over the entire horizon. We show by comparing to ground truth data that we achieve similar performance for full-body motion predictions as using oracle grasp and place locations.
BibTeX
@inproceedings{kratzer2020anticipating,
  author = {Philipp Kratzer and Niteesh Balachandra Midlagajni and Marc Toussaint and Jim Mainprice},
  title = {{Anticipating Human Intention for Full-Body Motion Prediction in Object Grasping and Placing Tasks}},
  booktitle = {RO-MAN},
  year = {2020},
  url = {https://openreview.net/forum?id=Fm5rQocJ_i},
}
Figure