Introduction
For robots to work safely and efficiently alongside humans, they must be able to "see into the future" to avoid collisions and coordinate tasks. While short-term motion prediction (the next few hundred milliseconds) is relatively well-handled by standard recurrent neural networks (RNNs), long-term prediction remains a significant challenge. Human movement is not merely a sequence of joint angles; it is driven by intent and constrained by the environment. For instance, a person reaching for a cup has a trajectory determined by the cup's location and available clear space on a table. Most existing models ignore this environmental context, leading to "uninformed" predictions that drift into obstacles or fail to reach the intended goal.
This research, presented at ICRA 2020, proposes a framework that predicts human motion by first anticipating "intent" through environmental affordances—the possible actions a scene allows. By combining deep learning (RNNs and CNNs) with trajectory optimization, the system forecasts complex, full-body manipulation movements that respect the 3D geometry of the environment.
Methods: The Offline-Online Pipeline
The researchers developed a two-stage pipeline consisting of an offline training phase and an online prediction phase. The offline phase involves training affordance models (for graspability and placeability) and a full-body motion prediction RNN using a large motion database.
Affordance Modeling
The system identifies "where" an action might occur:
- Placeability: A Mixture Density Network (MDN) combined with a CNN processes 2D occupancy maps and Signed Distance Fields (SDF) of surfaces to identify valid placement zones. The MDN allows the model to represent multiple possible goals (multi-modality), acknowledging that a person could place an object in several different spots.
- Graspability: The likelihood of wrist positions around an object is modeled using both Gaussian and von Mises-Fisher (vMF) distributions, the latter being particularly suited for modeling points on spherical surfaces.
Motion Prediction and Optimization
The system then determines "how" the motion is performed:
- Trajectory Generation: A VRED (Position-Velocity Recurrent Encoder-Decoder) network generates a base kinematic trajectory.
- Trajectory Optimization: At runtime, the system samples a goal from the affordance model and uses a non-linear optimizer (L-BFGS) to refine the VRED's predicted trajectory. This ensures the final motion is kinematically natural while strictly satisfying the "goal constraint" (e.g., the hand actually ends up at the predicted grasp point).
Experimental Setup
The authors collected a custom dataset using an Optitrack motion capture system in a $4 \times 4$ meter area. Five subjects wearing 50-marker MoCap suits performed 1,551 grasp-place sequences, such as setting and clearing a table with cups, jugs, plates, and bowls. The environment included a table, a large shelf, and a small shelf to provide varied heights and surfaces.
Key Insights
- Geometry as a First-Class Citizen: By incorporating Signed Distance Fields (SDFs) into the affordance model, the system learns to avoid placing objects where they would overlap with existing ones.
- The Power of Multi-modality: Using MDNs is crucial. A standard regressor might predict the "average" of all valid spots, which often lands in the middle of an obstacle. MDNs maintain distinct "peaks" of probability for different valid regions.
- Transfer Learning for Spatial Awareness: Pre-training an autoencoder to reconstruct occupancy maps helped the "Placeability" network learn more robust features about the environment's layout, leading to a 5% improvement in predicting valid placement regions.
- Optimization as a Bridge: Purely data-driven RNNs often "miss" the goal slightly. Using the RNN as a prior for a trajectory optimizer provides the best of both worlds: natural-looking motion that actually completes the task.
Results
The proposed method significantly outperformed baselines in both affordance accuracy and full-body motion prediction.
- Affordance Accuracy: The "Transfer Learning + Penalty" model for placeability significantly outperformed baselines, especially when the human was still far from the table (4 seconds before placement).
- Grasping: Both neural models (Gaussian and vMF) beat the heuristic baseline by a wide margin. While the Gaussian model had slightly lower mean squared error, the vMF model provided more useful directional uncertainty for the approach angle.
- Full-Body Prediction: When predicting 1.5 seconds into the future, the proposed method achieved an error rate nearly identical to an "oracle" (a model given the true ground-truth goal). It significantly outperformed a standard VRED RNN, which tended to "undershoot" the movement.
Limitations and Future Work
Current limitations include the small participant pool (5 subjects) and the focus on immediate next-action prediction. Future work could integrate higher-level symbolic planning to predict sequences of actions (e.g., "pick up the bottle, then pour into the glass"). Additionally, transitioning the system to work with standard RGB-D cameras in real-time is a necessary step for practical robotics.
Summary updated