Online Vehicle Trajectory Prediction using Policy Anticipation Network and Optimization-based Context Reasoning

Wenchao Ding, Shaojie Shen

Introduction

Predicting vehicle trajectories in dense urban environments is a critical challenge for autonomous driving. Two primary factors complicate this task: multimodality and complex environmental context. At intersections, vehicles often face multiple valid paths (e.g., turning left vs. going straight), and their movement is strictly governed by lane geometries, traffic signals, and the presence of other agents. Standard regression models often fail to capture these discrete shifts in intent, sometimes predicting physically impossible "average" paths.

Visualization of the complex urban environment in the CARLA simulator, highlighting various contextual factors like traffic signs, road construction, and moving agents.

Proposed Framework

To address these challenges, this paper introduces a hierarchical "Online Two-Level" framework that mimics human driving logic by separating high-level intent from low-level physical execution.

The proposed two-level framework: a Policy Anticipation Network for high-level intent and Optimization-based Context Reasoning for low-level trajectory refinement.

Level 1: High-Level Policy Anticipation

The first stage utilizes an RNN-based encoder-decoder (LSTM) to process a vehicle's history and predict a discrete driving policy, such as forward, yield, or turn left. By identifying the most likely intention first, the system effectively prunes the search space and avoids the pitfalls of multimodal averaging. This predicted policy is then mapped to the local environment to generate a rough "reference trajectory."

Level 2: Low-Level Context Reasoning

The second stage employs non-linear optimization (via the Ceres solver) to refine the reference trajectory into a smooth, feasible path. This process utilizes a multi-layer cost map that accounts for:

  • Static Constraints: Lane boundaries and physical obstacles.
  • Dynamic Constraints: Predicted positions of other moving objects (MO).
  • Traffic Regulations: Soft constraints for rules like red lights, allowing the model to predict trajectories even for law-breaking vehicles.
  • Nonholonomic Constraints: The physical limits of vehicle acceleration and curvature.

Experimental Setup and Results

The framework was evaluated using the CARLA autonomous driving simulator (v0.7.0). The researchers collected over 21,000 frames of data, including both standard driving and traffic violations, to ensure model robustness. Both observation and prediction horizons were set to 4 seconds.

Key results include:

  • Significant Accuracy Gains: The proposed method achieved a 44% accuracy improvement on curved roads and a 56% improvement at intersections compared to standard RNN baselines.
  • Context Sensitivity: The model successfully predicted "hard braking" for pedestrians, a scenario where baseline models often incorrectly predicted continued forward motion.
  • Real-Time Performance: The entire pipeline—including inference, cost map rendering, and optimization—averages 22.6 ms, making it highly suitable for online deployment.

Conclusion and Future Work

By treating trajectory prediction as an optimization problem rather than a black-box regression, this framework provides a transparent and flexible way to incorporate complex environmental rules. Future work aims to replace manual weight tuning for cost terms with Inverse Reinforcement Learning (IRL) and to further explore complex multi-agent interactions where vehicles' intentions are mutually influential.

Summary updated

Abstract

In this paper, we present an online two-level vehicle trajectory prediction framework for urban autonomous driving where there are complex contextual factors, such as lane geometries, road constructions, traffic regulations and moving agents. Our method combines high-level policy anticipation with low-level context reasoning. We leverage a long short-term memory (LSTM) network to anticipate the vehicle's driving policy (e.g., forward, yield, turn left, turn right, etc.) using its sequential history observations. The policy is then used to guide a low-level optimization-based context reasoning process. We show that it is essential to incorporate the prior policy anticipation due to the multimodal nature of the future trajectory. Moreover, contrary to existing regression-based trajectory prediction methods, our optimization-based reasoning process can cope with complex contextual factors. The final output of the two-level reasoning process is a continuous trajectory that automatically adapts to different traffic configurations and accurately predicts future vehicle motions. The performance of the proposed framework is analyzed and validated in an emerging autonomous driving simulation platform (CARLA).
BibTeX
@inproceedings{ding2019online,
  author = {Wenchao Ding and Shaojie Shen},
  title = {{Online Vehicle Trajectory Prediction using Policy Anticipation Network and Optimization-based Context Reasoning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=gmwYFYu1ytdL},
}
Figure