TL;DR
This paper introduces RobustTP, an end-to-end trajectory prediction system for dense, heterogeneous traffic using noisy sensor inputs from RGB cameras. It achieves state-of-the-art performance by leveraging a novel tracking-by-detection pipeline and an LSTM-CNN architecture that models agent interactions, outperforming prior methods in average and final displacement error.
What Problem Does This Paper Address?
Predicting the future movements of road agents (cars, pedestrians, scooters, etc.) is crucial for advanced driver-assistance systems (ADAS) and autonomous driving. However, accurately predicting trajectories in dense, heterogeneous traffic—where many different types of agents with varying dynamics interact closely—is a significant challenge. Existing methods often rely on manually annotated trajectories for training, which are expensive to obtain, may not reflect real-world complexities, and can introduce dataset bias. Furthermore, real-world sensor inputs (like those from cameras) are inherently noisy due to tracking inaccuracies. This paper aims to bridge this gap by developing a robust, end-to-end system that can predict trajectories using noisy, sensor-derived inputs in challenging traffic conditions.
Methods
RobustTP employs a two-stage approach:
- Tracking-by-Detection: Instead of relying on manually labeled ground truth, RobustTP uses a tracking algorithm to generate input trajectories from raw RGB video. This pipeline starts with instance segmentation (using Mask R-CNN) to create occlusion-free representations of road agents. These segmented agents are then tracked using a modified Reciprocal Velocity Obstacles (RVO) formulation, which accounts for collision avoidance in dense environments. The RVO formulation is adapted to work with segmented agents and predicts their next state. This process generates noisy trajectory histories that are then fed into the prediction module.
- Trajectory Prediction: The noisy trajectory histories are used to train a deep learning model. RobustTP utilizes an LSTM-CNN neural network architecture. This architecture is designed to model interactions between heterogeneous road agents. The LSTM component handles temporal dependencies in the trajectories, while the CNN component can capture spatial relationships and local features. This combined approach allows the model to learn complex interactions in dense traffic.
The authors also introduce TrackNPred, a software framework that implements various tracking and trajectory prediction methods, including RobustTP, to facilitate benchmarking and evaluation on real-world datasets.
Datasets
The primary dataset used for training and evaluation is the TRAF dataset. This dataset features dense and heterogeneous traffic scenarios captured by static or moving RGB cameras. It includes various road agent categories (cars, buses, scooters, pedestrians, etc.) and is categorized by camera viewpoint, motion, time of day, and traffic density. The paper notes that the TRAF dataset contains approximately 13 motorized vehicles, 5 pedestrians, and 2 bicycles per frame.
Key Insights and Ideas
- Noisy Inputs are Sufficient: A key insight is that robust trajectory prediction can be achieved even with noisy trajectory histories derived from sensor inputs, rather than requiring perfect, manually annotated data. This significantly lowers the barrier to real-world deployment.
- Instance Segmentation for Robust Tracking: Using instance segmentation (Mask R-CNN) instead of simple bounding boxes proves crucial for handling occlusions and generating better representations of agents in dense traffic, leading to more reliable tracking.
- RVO for Dense Traffic Interaction: Adapting the RVO formulation for collision avoidance allows the tracking module to better handle the complex interactions and potential collisions in dense traffic scenarios.
- LSTM-CNN for Heterogeneous Interactions: The combination of LSTMs and CNNs effectively models both temporal dynamics and spatial interactions between diverse road agents, which is critical for accurate prediction in heterogeneous traffic.
- TrackNPred as a Benchmarking Tool: The release of TrackNPred is significant, providing a standardized platform for comparing and developing future trajectory prediction models, especially for real-world, dense traffic scenarios.
Key Results
RobustTP demonstrates significant improvements over state-of-the-art methods:
- Average Displacement Error (ADE): Up to an 18% improvement compared to the next best method.
- Final Displacement Error (FDE): Up to a 35.5% improvement at the end of the 5-second prediction window.
- When trained on noisy sensor inputs, RobustTP consistently achieves lower RMSE curves than other methods after the first second of prediction, indicating better longer-term performance.
- The final RMSE for RobustTP is reported to be well below 4 meters after 5 seconds, which is considered a crucial metric for reliability in real-world applications.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the size of the TRAF dataset limits the training of larger deep learning networks, potentially affecting generalization to all forms of dense and heterogeneous traffic.
- ADAS Integration: RobustTP is presented as a proof of concept for ADAS. Future work includes designing it as an effective ADAS, leveraging its computational efficiency and ability to generalize across varying dynamics.
- Online Hard Example Mining (OHEM): The authors plan to explore OHEM to improve generalization to real-world scenarios, especially for autonomous driving applications.
- Framework Expansion: They also aim to expand the TrackNPred framework by adding more algorithms and hyperparameter tuning capabilities.
Summary updated