This paper, titled "Heterogeneous Trajectory Forecasting via Risk and Scene Graph Learning," presents a sophisticated approach to one of the most difficult problems in autonomous driving: predicting the future movements of a diverse set of road users—cars, pedestrians, and cyclists—in complex urban environments.
TL;DR
The authors propose a novel forecasting framework that explicitly models the interplay between different types of road agents and their environment using two specialized graphs: a Heterogeneous Risk Graph (HRG) and a Hierarchical Scene Graph (HSG). By combining collision risk reasoning with a structured "grammar" of the road scene, the model achieves state-of-the-art performance on major benchmarks like nuScenes and Argoverse.
What Problem Does This Paper Address?
Predicting where a car will be in three seconds is hard; predicting where a car, a group of pedestrians, and a cyclist will be—while they all interact with each other and follow (or break) traffic rules—is significantly harder.
Current methods often struggle with two things:
- Heterogeneity: Pedestrians, cars, and cyclists have vastly different movement patterns and social "rules" of interaction.
- Environmental Constraints: While many models use semantic maps (e.g., "this pixel is a sidewalk"), they often fail to capture the logical and topological constraints of the scene, such as the fact that a car shouldn't suddenly teleport onto a sidewalk, or that a crosswalk naturally links two sides of a road.
Methods
The core of the proposed method is the dual-graph architecture, illustrated in the main architecture diagram:
1. Heterogeneous Risk Graph (HRG)
Instead of just looking at the physical distance between agents, the HRG calculates an interaction adjacency matrix based on collision risk. This risk metric considers:
- Agent Attributes: Location, velocity, and heading.
- Moving Patterns: The authors use spectral clustering on historical trajectories to group agents by their behavior (e.g., "straight walking" vs. "turning").
- Time to Collision (TTC): A dynamic measure of how soon two agents might collide.
- Moving Direction Risk: Whether an agent is in another's forward field of view.
2. Hierarchical Scene Graph (HSG)
The HSG moves beyond raw pixels to model the road as a structured "grammar." It builds a tree-like hierarchy of the scene (e.g., Road -> Drivable Area -> Lane).
By aligning agents to these semantic regions, the model learns the natural constraints of the environment. For example, a pedestrian on a sidewalk has zero collision risk with a car in a lane, but that risk "activates" the moment their paths converge on a crosswalk (Zebra region).
3. Fusion and Decoding
These two graphs are processed through stacked Graph Convolutional Networks (GCN) for spatial relations and Temporal Convolutional Networks (TCN) for time-series evolution. The resulting embeddings are fused via a dot-product operation and decoded into a Bivariate Gaussian Probability Distribution, which allows the model to output not just a single path, but a distribution representing the uncertainty of future movements.
Datasets
The model was evaluated on three industry-standard datasets:
- nuScenes: A large-scale dataset with diverse agents (cars, peds, etc.) and high-definition (HD) semantic maps.
- Argoverse: Focused primarily on vehicle trajectories with rich HD maps.
- ApolloScape: A massive dataset, though it lacks the HD maps required to build the full HSG, allowing the authors to test the HRG component in isolation.
Key Insights and Ideas
- Risk as the Primary Interaction Metric: Moving from simple Euclidean distance to a multi-factor collision risk metric (incorporating TTC and heading) provides a much more meaningful representation of how agents influence one another.
- Grammar-Based Scene Understanding: Explicitly modeling the "rules" of the road (e.g., sidewalks are for walking, lanes are for driving) acts as a powerful prior that prevents the model from predicting physically impossible or highly illegal trajectories.
- The Power of Fusion: The authors found that a dot-product fusion of the Risk and Scene graphs significantly outperformed simple residual connections. This suggests that the scene structure provides a critical "mask" or constraint that refines the raw risk calculations.
Key Results
The proposed method, particularly the HRG+HSG-dot version, consistently outperformed or matched state-of-the-art models across multiple metrics (mADE, mFDE).
- nuScenes (@3s): Achieved an mADE of 0.930m and mFDE of 1.320m, representing a 9.18% improvement in mADE and a 25.8% improvement in mFDE over the well-known DESIRE* model.
- Argoverse: Achieved an mADE of 0.850m and mFDE of 1.123m, showing strong performance even in vehicle-only scenarios.
- Explainability: Qualitative results show that the model correctly identifies high-risk moments, such as two cars merging into the same lane, which are clearly reflected in the HRG's risk matrix.
Limitations and Future Work
- Map Dependency: The Hierarchical Scene Graph (HSG) relies heavily on high-quality, pre-annotated HD maps. As seen in the ApolloScape tests, the model's performance gains are more modest when these maps are unavailable.
- Sampling Fluctuations: Because the model outputs a probability distribution, individual sampled trajectories can sometimes be "jittery." The authors addressed this using Bezier Curves to smooth the final output, but more integrated kinematic constraints could be explored.
- Future Directions: The authors aim to incorporate moving intentions (e.g., why is that pedestrian walking toward the curb?) and explore the model's utility in predicting actual traffic accidents and performing counterfactual analysis (e.g., "what would have happened if the car hadn't braked?").
Summary updated