Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction

Ajay Jain, Sergio Casas, Renjie Liao, Yuwen Xiong, Song Feng, Sean Segal, Raquel Urtasun

Abstract

Self-driving vehicles plan around both static and dynamic objects, applying predictive models of behavior to estimate future locations of the objects in the environment. However, future behavior is inherently uncertain, and models of motion that produce deterministic outputs are limited to short timescales. Particularly difficult is the prediction of human behavior. In this work, we propose the discrete residual flow network (DRF-Net), a convolutional neural network for human motion prediction that captures the uncertainty inherent in long-range motion forecasting. In particular, our learned network effectively captures multimodal posteriors over future human motion by predicting and updating a discretized distribution over spatial locations. We compare our model against several strong competitors and show that our model outperforms all baselines.

Introduction: The Challenge of Pedestrian Prediction

Predicting pedestrian movement is a critical yet difficult task for autonomous systems. Humans are inherently unpredictable; we may stop abruptly, change direction to avoid obstacles, or choose between multiple equally valid paths. This "multimodality"—where several distinct future outcomes are possible—poses a significant challenge for traditional models.

Multimodal pedestrian behavior prediction: (a) belief over future states, (b) evolution of probability over time, and (c) environmental context like crosswalks.

Most existing models attempt to predict a single "best" trajectory or a small set of samples. However, models minimizing Mean Squared Error (MSE) often converge on a "conditional average"—a path that might physically pass through a wall or a parked car because it represents the mathematical midpoint between two valid routes. While Gaussian distributions can handle some uncertainty, real-world human behavior is rarely "bell-shaped." DRF-NET addresses this by representing complex, non-Gaussian possibilities over long time horizons (up to 10 seconds).

The DRF-NET Architecture

The core innovation of the Discrete Residual Flow Network (DRF-NET) is the shift from predicting specific $(x, y)$ coordinates to predicting a Probability Mass Function (PMF) over a discretized spatial grid. This allows the model to represent arbitrary shapes of uncertainty that can "flow" around obstacles.

The DRF-Net architecture, showing scene rasterization, feature extraction via FPN, and the recursive discrete residual flow head producing beliefs over future states.

Methodology

  1. Scene Rasterization: The model encodes a bird’s-eye view of the environment, including map features (crosswalks, lanes), traffic light states, and the history of dynamic objects into a multi-channel tensor.
  2. Backbone & FPN: A ResNet-18 backbone and Feature Pyramid Network (FPN) extract high-level features from the rasterized scene.
  3. Discrete Residual Flow: Instead of independent frame predictions, the model recursively updates the probability distribution. It takes the log-probability of the pedestrian's location at time $t-1$ and adds a predicted "residual" to produce the distribution for time $t$. This functions like a learned Markov chain conditioned on the map and history.
  4. ModePool Operator: To evaluate multimodality, the authors developed "ModePool," a GPU-efficient operator that identifies local maxima in the probability grid to count distinct predicted paths.

Datasets and Evaluation

The study utilized a massive dataset of 481,927 ground-truth pedestrian trajectories from several North American cities. The data includes 6 seconds of history and 10 seconds of future ground truth. To ensure robustness, the model was also evaluated against "online" tracks—noisy data from actual on-vehicle object detection systems—to test its performance under sensor errors and occlusions.

Key Results and Insights

DRF-NET significantly outperformed baselines such as Mixture Density Networks (MDNs) and ConvLSTMs across several metrics:

  • Likelihood: DRF-NET achieved a 90% increase in geometric mean likelihood compared to the best MDN baseline on ground-truth tracks.
  • Accuracy: For a 10-second horizon, DRF-NET achieved a Final Displacement Error (FDE) of 2.71m, significantly better than ConvLSTM (3.20m) and standard Density Networks (7.66m).
  • Temporal Consistency: By using residual flow to update previous beliefs, the model maintains better temporal logic. Uncertainty grows naturally over time as the "flow" spreads out, capturing distinct modes like stopping, crossing, or skirting obstacles.

Conclusion

By treating the world as a grid (0.5m bins), DRF-NET avoids the limitations of parametric distributions. While there are challenges regarding computational costs for high-resolution grids and the need for deeper intent modeling (e.g., internal traffic light states), the discrete flow approach provides a much more useful occupancy grid for a self-driving car's motion planner to calculate risk in complex urban environments.

Summary updated

Abstract

Self-driving vehicles plan around both static and dynamic objects, applying predictive models of behavior to estimate future locations of the objects in the environment. However, future behavior is inherently uncertain, and models of motion that produce deterministic outputs are limited to short timescales. Particularly difficult is the prediction of human behavior. In this work, we propose the discrete residual flow network (DRF-Net), a convolutional neural network for human motion prediction that captures the uncertainty inherent in long-range motion forecasting. In particular, our learned network effectively captures multimodal posteriors over future human motion by predicting and updating a discretized distribution over spatial locations. We compare our model against several strong competitors and show that our model outperforms all baselines.
BibTeX
@inproceedings{jain2019discrete,
  author = {Ajay Jain and Sergio Casas and Renjie Liao and Yuwen Xiong and Song Feng and Sean Segal and Raquel Urtasun},
  title = {{Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=i0LAW2PyCfg},
}
Figure