RobustTP: End-to-End Trajectory Prediction for Heterogeneous Road-Agents in Dense Traffic with Noisy Sensor Inputs

Rohan Chandra, Uttaran Bhattacharya, Christian Roncal, Aniket Bera, Dinesh Manocha

TL;DR

This paper introduces RobustTP, an end-to-end trajectory prediction system for dense, heterogeneous traffic using noisy sensor inputs from RGB cameras. It achieves state-of-the-art performance by leveraging a novel tracking-by-detection pipeline and an LSTM-CNN architecture that models agent interactions, outperforming prior methods in average and final displacement error.

What Problem Does This Paper Address?

Predicting the future movements of road agents (cars, pedestrians, scooters, etc.) is crucial for advanced driver-assistance systems (ADAS) and autonomous driving. However, accurately predicting trajectories in dense, heterogeneous traffic—where many different types of agents with varying dynamics interact closely—is a significant challenge. Existing methods often rely on manually annotated trajectories for training, which are expensive to obtain, may not reflect real-world complexities, and can introduce dataset bias. Furthermore, real-world sensor inputs (like those from cameras) are inherently noisy due to tracking inaccuracies. This paper aims to bridge this gap by developing a robust, end-to-end system that can predict trajectories using noisy, sensor-derived inputs in challenging traffic conditions.

Methods

RobustTP employs a two-stage approach:

  1. Tracking-by-Detection: Instead of relying on manually labeled ground truth, RobustTP uses a tracking algorithm to generate input trajectories from raw RGB video. This pipeline starts with instance segmentation (using Mask R-CNN) to create occlusion-free representations of road agents. These segmented agents are then tracked using a modified Reciprocal Velocity Obstacles (RVO) formulation, which accounts for collision avoidance in dense environments. The RVO formulation is adapted to work with segmented agents and predicts their next state. This process generates noisy trajectory histories that are then fed into the prediction module.
  2. Trajectory Prediction: The noisy trajectory histories are used to train a deep learning model. RobustTP utilizes an LSTM-CNN neural network architecture. This architecture is designed to model interactions between heterogeneous road agents. The LSTM component handles temporal dependencies in the trajectories, while the CNN component can capture spatial relationships and local features. This combined approach allows the model to learn complex interactions in dense traffic.

The authors also introduce TrackNPred, a software framework that implements various tracking and trajectory prediction methods, including RobustTP, to facilitate benchmarking and evaluation on real-world datasets.

Datasets

The primary dataset used for training and evaluation is the TRAF dataset. This dataset features dense and heterogeneous traffic scenarios captured by static or moving RGB cameras. It includes various road agent categories (cars, buses, scooters, pedestrians, etc.) and is categorized by camera viewpoint, motion, time of day, and traffic density. The paper notes that the TRAF dataset contains approximately 13 motorized vehicles, 5 pedestrians, and 2 bicycles per frame.

Key Insights and Ideas

  • Noisy Inputs are Sufficient: A key insight is that robust trajectory prediction can be achieved even with noisy trajectory histories derived from sensor inputs, rather than requiring perfect, manually annotated data. This significantly lowers the barrier to real-world deployment.
  • Instance Segmentation for Robust Tracking: Using instance segmentation (Mask R-CNN) instead of simple bounding boxes proves crucial for handling occlusions and generating better representations of agents in dense traffic, leading to more reliable tracking.
  • RVO for Dense Traffic Interaction: Adapting the RVO formulation for collision avoidance allows the tracking module to better handle the complex interactions and potential collisions in dense traffic scenarios.
  • LSTM-CNN for Heterogeneous Interactions: The combination of LSTMs and CNNs effectively models both temporal dynamics and spatial interactions between diverse road agents, which is critical for accurate prediction in heterogeneous traffic.
  • TrackNPred as a Benchmarking Tool: The release of TrackNPred is significant, providing a standardized platform for comparing and developing future trajectory prediction models, especially for real-world, dense traffic scenarios.

Key Results

RobustTP demonstrates significant improvements over state-of-the-art methods:

  • Average Displacement Error (ADE): Up to an 18% improvement compared to the next best method.
  • Final Displacement Error (FDE): Up to a 35.5% improvement at the end of the 5-second prediction window.
  • When trained on noisy sensor inputs, RobustTP consistently achieves lower RMSE curves than other methods after the first second of prediction, indicating better longer-term performance.
  • The final RMSE for RobustTP is reported to be well below 4 meters after 5 seconds, which is considered a crucial metric for reliability in real-world applications.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the size of the TRAF dataset limits the training of larger deep learning networks, potentially affecting generalization to all forms of dense and heterogeneous traffic.
  • ADAS Integration: RobustTP is presented as a proof of concept for ADAS. Future work includes designing it as an effective ADAS, leveraging its computational efficiency and ability to generalize across varying dynamics.
  • Online Hard Example Mining (OHEM): The authors plan to explore OHEM to improve generalization to real-world scenarios, especially for autonomous driving applications.
  • Framework Expansion: They also aim to expand the TrackNPred framework by adding more algorithms and hyperparameter tuning capabilities.

Summary updated

Abstract

We present RobustTP, an end-to-end algorithm for predicting future trajectories of road-agents in dense traffic with noisy sensor input trajectories obtained from RGB cameras (either static or moving) through a tracking algorithm. In this case, we consider noise as the deviation from the ground truth trajectory. The amount of noise depends on the accuracy of the tracking algorithm. Our approach is designed for dense heterogeneous traffic, where the road agents corresponding to a mixture of buses, cars, scooters, bicycles, or pedestrians. RobustTP is an approach that first computes trajectories using a combination of a non-linear motion model and a deep learning-based instance segmentation algorithm. Next, these noisy trajectories are trained using an LSTM-CNN neural network architecture that models the interactions between road-agents in dense and heterogeneous traffic. Our trajectory prediction algorithm outperforms state-of-the-art methods for end-to-end trajectory prediction using sensor inputs. We achieve an improvement of upto 18% in average displacement error and an improvement ofup to 35.5% in final displacement error at the end of the prediction window (5 seconds) over the next best method. All experiments were set up on an Nvidia TiTan Xp GPU. Additionally, we release a software framework, TrackNPred. The framework consists of implementations of state-of-the-art tracking and trajectory prediction methods and tools to benchmark and evaluate them on real-world dense traffic datasets.
BibTeX
@inproceedings{chandra2019robusttp,
  author = {Rohan Chandra and Uttaran Bhattacharya and Christian Roncal and Aniket Bera and Dinesh Manocha},
  title = {{RobustTP: End-to-End Trajectory Prediction for Heterogeneous Road-Agents in Dense Traffic with Noisy Sensor Inputs}},
  booktitle = {CSCS},
  year = {2019},
  doi = {10.1145/3359999.3360495},
}
Figure