Predicting where a pedestrian will be in five seconds is notoriously difficult because humans don't just follow physics; they follow intentions. This paper introduces a monolithic deep neural network that emulates human planning by first inferring potential destinations and then "planning" a path toward them, significantly improving long-term prediction accuracy for autonomous vehicles.
TL;DR
The authors propose an end-to-end neural network that predicts pedestrian motion by emulating their internal planning process. By first inferring a pedestrian's destination and then "planning" a trajectory toward it, the system achieves much higher accuracy over long time horizons than traditional physics-based filters.
What Problem Does This Paper Address?
For autonomous vehicles to navigate safely, they must anticipate the movement of "vulnerable road users" like pedestrians. Traditional methods often rely on simple physics-based extrapolation (e.g., Kalman filters), which works well for the next half-second but fails over longer horizons because pedestrians frequently change direction to reach goals or avoid obstacles.
Existing goal-based models often require researchers to manually define possible destinations or work only in static environments. The authors of this paper aim to bridge this gap by creating a system that learns to recognize intentions and plan trajectories directly from raw sensor data (images and positions) in dynamic, real-world scenes.
Methods
The authors propose a single, end-to-end trainable architecture composed of three interconnected sub-networks:
- Destination Network: This uses a Recurrent Mixture Density Network (RMDN)—an LSTM combined with a CNN—to process pedestrian images and past positions. It outputs a probability map of potential destinations, modeled as a mixture of Gaussian-von-Mises distributions.
- Topology Network: A Fully Convolutional Network (FCN) takes the predicted destinations and environment features (like road and sidewalk maps) to generate a "topology map." This map acts as a cost or reward surface for the planning stage.
- Planning Network: This stage simulates the pedestrian's internal decision-making. The authors tested two approaches:
- Markov Decision Process (MDP): Uses Value Iteration, implemented as recursive CNN layers, to find an optimal policy.
- Forward-Backward Algorithm: A probabilistic approach that propagates probabilities from the start to the future and from the goal back to the past.
The entire system is trained using Inverse Reinforcement Learning (IRL), where the network learns the "reward" function (the topology map) that makes the observed human trajectories look optimal.
Datasets
The model was trained and evaluated on a custom dataset of stereo videos collected from urban and residential areas.
- Size: Approximately 400 individual pedestrian tracks.
- Features: Semantic segmentation (obstacles, road, sidewalk) was performed using a GoogleNet trained on the Cityscapes dataset.
- Ground Truth: Pedestrian trajectories were manually annotated and optimized using a constant velocity model for offline processing.
Key Insights and Ideas
- Planning as a Layer: By formulating classical planning algorithms (like Value Iteration) as differentiable neural network layers, the authors can backpropagate through the entire planning process. This allows the model to learn why a pedestrian chose a certain path.
- Intention Matters: The "aha moment" is that predicting the destination first makes the trajectory prediction much more stable. While a standard filter's error grows linearly with time, the Destination Network's accuracy degrades much more slowly (sub-linearly).
- Implicit Obstacle Avoidance: Because the Topology Network is trained to match human behavior, it naturally learns that obstacles have low "reward," causing the Planning Network to route trajectories around them without needing explicit hand-coded rules.
Key Results
- Long-term Stability: Compared to a standard Interacting Multiple Model (IMM) filter, the proposed RMDN-based approach maintains much higher prediction certainty over a 3-second horizon.
- MDP vs. Forward-Backward: The MDP-based planning network performed slightly better than the Forward-Backward approach, though both were significantly more accurate than baseline filters.
- Qualitative Success: The model successfully predicts paths that curve around obstacles, a behavior that standard recursive filters cannot capture.
Limitations and Future Work
- Map Dependency: The current system relies on precomputed semantic maps. The authors suggest that future versions could learn to extract these features directly from raw sensor data without manual annotation.
- Discretization: The planning happens on a grid map, which introduces quantization errors and limits the precision of the predicted movements.
- Dataset Size: With only 400 tracks, the model's ability to generalize to highly complex or rare scenarios remains to be tested on larger, more diverse datasets.
Summary updated