Abstract
Predicting pedestrian motion is a critical challenge for autonomous systems due to the intentional nature of human movement. This paper presents an end-to-end deep neural network architecture that models pedestrian behavior as a planning process. By sequentially inferring destinations, generating environmental cost maps, and simulating path planning, the proposed system achieves superior long-term prediction accuracy compared to traditional physics-based filters.
Introduction
For autonomous vehicles to navigate safely, they must anticipate the movement of "vulnerable road users" like pedestrians. Traditional methods often rely on simple physics-based extrapolation, such as Kalman filters. While effective for short-term horizons (e.g., 0.5 seconds), these methods fail over longer periods because pedestrians frequently change direction to reach specific goals or avoid environmental obstacles.
Existing goal-based models often require manually defined destinations or are restricted to static environments. This research bridges the gap by introducing a system that learns to recognize intentions and plan trajectories directly from raw sensor data—specifically images and positions—in dynamic, real-world scenes.
Methodology
The authors propose a monolithic, end-to-end trainable architecture composed of three interconnected sub-networks:
1. Destination Network
The process begins with a Recurrent Mixture Density Network (RMDN), which combines an LSTM with a CNN to process pedestrian images and historical positions. This network outputs a probability map of potential destinations, modeled as a mixture of Gaussian-von-Mises distributions.
2. Topology Network
A Fully Convolutional Network (FCN) takes the predicted destinations and environment features (such as road and sidewalk maps) to generate a "topology map." This map serves as a cost or reward surface for the subsequent planning stage.
3. Planning Network
This stage simulates the pedestrian's internal decision-making. The authors evaluated two distinct approaches:
- Markov Decision Process (MDP): Utilizes Value Iteration, implemented as recursive CNN layers, to derive an optimal policy.
- Forward-Backward Algorithm: A probabilistic approach that propagates probabilities from the start to the future and from the goal back to the past.
The entire system is trained using Inverse Reinforcement Learning (IRL). In this framework, the network learns the "reward" function (the topology map) that makes observed human trajectories appear optimal.
Experimental Setup
The model was trained and evaluated on a custom dataset of stereo videos collected from urban and residential areas, comprising approximately 400 individual pedestrian tracks. Semantic segmentation (identifying obstacles, roads, and sidewalks) was performed using a GoogleNet pre-trained on the Cityscapes dataset. Ground truth trajectories were manually annotated and optimized using a constant velocity model for offline processing.
Key Insights
- Planning as a Differentiable Layer: By formulating classical planning algorithms like Value Iteration as differentiable neural network layers, the model can backpropagate through the entire planning process. This allows the system to learn the underlying motivations behind pedestrian paths.
- Intention-Driven Stability: A key finding is that predicting the destination first stabilizes trajectory prediction. While standard filter errors grow linearly with time, the Destination Network's accuracy degrades sub-linearly, providing much more reliable long-term forecasts.
- Implicit Environmental Awareness: Because the Topology Network is trained to match human behavior, it naturally learns that obstacles have low "reward." Consequently, the Planning Network routes trajectories around them without requiring explicit, hand-coded rules.
Results and Discussion
The proposed RMDN-based approach maintains significantly higher prediction certainty over a 3-second horizon compared to standard Interacting Multiple Model (IMM) filters. While the MDP-based planning network performed slightly better than the Forward-Backward approach, both outperformed baseline filters.
Qualitatively, the model successfully predicts paths that curve around obstacles—a behavior that standard recursive filters cannot capture. This demonstrates the power of integrating planning into the deep learning pipeline.
Limitations and Future Work
Despite its success, the system has limitations. It currently relies on precomputed semantic maps; future iterations could aim to extract these features directly from raw sensor data. Additionally, the planning occurs on a grid map, which introduces quantization errors. Finally, expanding the dataset beyond 400 tracks will be necessary to test the model's generalization to more complex or rare urban scenarios.
Summary updated