Abstract
Motion forecasting is an important component in autonomous driving systems. One of the most challenging problems in motion forecasting is interactive trajectory prediction, whose goal is to jointly forecasts the future trajectories of interacting agents. To this end, we present a large-scale interactive trajectory prediction dataset named INT2 for INTeractive trajectory prediction at INTersections. INT2 includes 612,000 scenes, each lasting 1 minute, containing up to 10,200 hours of data. The agent trajectories are auto-labeled by a high-performance offline temporal detection and fusion algorithm, whose quality is further inspected by human judges. Vectorized semantic maps and traffic light information are also included in INT2. Additionally, the dataset poses an interesting domain mismatch challenge. For each intersection, we treat rush-hour and non-rush-hour segments as different domains. We benchmark the best open-sourced interactive trajectory prediction method on INT2 and Waymo Open Motion, under in-domain and cross-domain settings.
Introduction
Predicting the future movement of vehicles is a cornerstone of autonomous driving. However, predicting how multiple agents will interact—reacting to one another in real-time—remains a significant challenge. Most current models treat agents as independent, often leading to "ghost collisions" in predicted futures. While datasets like the Waymo Open Motion Dataset (WOMD) have advanced the field, they are often limited by scale and a lack of focus on behavioral domain shifts.
The INT2 dataset addresses these gaps by providing a massive benchmark specifically for intersections, where complex "social dances" between drivers occur. Crucially, it introduces the concept of "temporal domain mismatch," where driving behavior changes fundamentally based on the time of day (e.g., rush hour vs. non-rush hour), even when the physical map remains identical.
Methods
The data was collected at 16 major intersections using a pole-mounted sensor suite comprising 12 cameras and 4 LiDARs.
Perception and Labeling
An offline perception pipeline was employed to generate high-fidelity ground truth. This system utilizes heavy-duty models like PointPillars for LiDAR processing and fuses inputs via a Kalman filter-based tracker. This approach allows for "production-level" accuracy that exceeds what is typically possible in real-time onboard systems.
Identifying Interactions
To isolate meaningful data, the authors developed a three-step interaction filtering process:
- Spatiotemporal Filtering: Identifying agent pairs in close proximity.
- Influencer-Reactor Labeling: Determining roles based on time-of-arrival at conflict points.
- Rule-Based Cleaning: Using human verification to ensure interactions are significant, such as a vehicle yielding to a pedestrian.
The INT2 Dataset
INT2 is approximately 10 times larger than previous industry standards like WOMD, containing 612,000 one-minute scenes and 106.8 million unique tracks.
A key feature of INT2 is its focus on behavioral variables. By keeping the map constant across 16 intersections, the dataset allows researchers to study how models generalize across different driving "moods"—from the aggressive, high-speed gaps of non-rush-hour traffic to the cautious, dense gridlock of rush hour. Furthermore, the 60-second scene length (compared to the standard 20 seconds) captures full traffic light cycles, providing essential context for agent behavior.
Results and Discussion
Benchmarking the state-of-the-art M2I predictor on INT2 revealed several key insights:
- Increased Complexity: M2I's performance was lower on INT2 than on WOMD, indicating that INT2's intersection scenarios are more challenging and diverse.
- Cross-Domain Performance Drop: Models trained on non-rush-hour data struggled significantly when tested on rush-hour data, and vice versa. For instance, the Miss Rate (MR) for vehicle-pedestrian interactions increased sharply in cross-domain settings.
- Qualitative Behavioral Shift: Models trained on "aggressive" non-rush-hour data tended to over-predict travel distances in "cautious" rush-hour scenarios, failing to account for the slower tempo of heavy traffic.
Conclusion
INT2 provides a massive, high-quality resource for studying interactive trajectory prediction. While currently focused on 16 intersections in a single city, it highlights the critical need for "domain-adaptive" predictors that can adjust to the varying "tempo" of human driving behavior. Future work will involve expanding geographic diversity and developing models that can better handle these temporal domain shifts.
For more information, visit the project repository: https://github.com/AIR-DISCOVER/INT2
Summary updated