Pedestrian Prediction by Planning Using Deep Neural Networks

Eike Rehder, Florian Wirth, Martin Lauer, Christoph Stiller

Predicting where a pedestrian will be in five seconds is notoriously difficult because humans don't just follow physics; they follow intentions. This paper introduces a monolithic deep neural network that emulates human planning by first inferring potential destinations and then "planning" a path toward them, significantly improving long-term prediction accuracy for autonomous vehicles.

TL;DR

The authors propose an end-to-end neural network that predicts pedestrian motion by emulating their internal planning process. By first inferring a pedestrian's destination and then "planning" a trajectory toward it, the system achieves much higher accuracy over long time horizons than traditional physics-based filters.

What Problem Does This Paper Address?

For autonomous vehicles to navigate safely, they must anticipate the movement of "vulnerable road users" like pedestrians. Traditional methods often rely on simple physics-based extrapolation (e.g., Kalman filters), which works well for the next half-second but fails over longer horizons because pedestrians frequently change direction to reach goals or avoid obstacles.

Existing goal-based models often require researchers to manually define possible destinations or work only in static environments. The authors of this paper aim to bridge this gap by creating a system that learns to recognize intentions and plan trajectories directly from raw sensor data (images and positions) in dynamic, real-world scenes.

Methods

The authors propose a single, end-to-end trainable architecture composed of three interconnected sub-networks:

  1. Destination Network: This uses a Recurrent Mixture Density Network (RMDN)—an LSTM combined with a CNN—to process pedestrian images and past positions. It outputs a probability map of potential destinations, modeled as a mixture of Gaussian-von-Mises distributions.
  2. Topology Network: A Fully Convolutional Network (FCN) takes the predicted destinations and environment features (like road and sidewalk maps) to generate a "topology map." This map acts as a cost or reward surface for the planning stage.
  3. Planning Network: This stage simulates the pedestrian's internal decision-making. The authors tested two approaches:
  4. Markov Decision Process (MDP): Uses Value Iteration, implemented as recursive CNN layers, to find an optimal policy.
  5. Forward-Backward Algorithm: A probabilistic approach that propagates probabilities from the start to the future and from the goal back to the past.
The complete prediction network architecture, showing the flow from raw image to final position probability maps.

The entire system is trained using Inverse Reinforcement Learning (IRL), where the network learns the "reward" function (the topology map) that makes the observed human trajectories look optimal.

Datasets

The model was trained and evaluated on a custom dataset of stereo videos collected from urban and residential areas.

  • Size: Approximately 400 individual pedestrian tracks.
  • Features: Semantic segmentation (obstacles, road, sidewalk) was performed using a GoogleNet trained on the Cityscapes dataset.
  • Ground Truth: Pedestrian trajectories were manually annotated and optimized using a constant velocity model for offline processing.

Key Insights and Ideas

  • Planning as a Layer: By formulating classical planning algorithms (like Value Iteration) as differentiable neural network layers, the authors can backpropagate through the entire planning process. This allows the model to learn why a pedestrian chose a certain path.
  • Intention Matters: The "aha moment" is that predicting the destination first makes the trajectory prediction much more stable. While a standard filter's error grows linearly with time, the Destination Network's accuracy degrades much more slowly (sub-linearly).
  • Implicit Obstacle Avoidance: Because the Topology Network is trained to match human behavior, it naturally learns that obstacles have low "reward," causing the Planning Network to route trajectories around them without needing explicit hand-coded rules.

Key Results

  • Long-term Stability: Compared to a standard Interacting Multiple Model (IMM) filter, the proposed RMDN-based approach maintains much higher prediction certainty over a 3-second horizon.
  • MDP vs. Forward-Backward: The MDP-based planning network performed slightly better than the Forward-Backward approach, though both were significantly more accurate than baseline filters.
  • Qualitative Success: The model successfully predicts paths that curve around obstacles, a behavior that standard recursive filters cannot capture.
An example of the input data: a camera image with tracked pedestrians.

Limitations and Future Work

  • Map Dependency: The current system relies on precomputed semantic maps. The authors suggest that future versions could learn to extract these features directly from raw sensor data without manual annotation.
  • Discretization: The planning happens on a grid map, which introduces quantization errors and limits the precision of the predicted movements.
  • Dataset Size: With only 400 tracks, the model's ability to generalize to highly complex or rare scenarios remains to be tested on larger, more diverse datasets.

Summary updated

Abstract

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
BibTeX
@inproceedings{rehder2018pedestrian,
  author = {Eike Rehder and Florian Wirth and Martin Lauer and Christoph Stiller},
  title = {{Pedestrian Prediction by Planning Using Deep Neural Networks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  doi = {10.1109/icra.2018.8460203},
}

Loading related content...

Figure