Abstract
Predicting the future trajectories of multiple interacting agents is a fundamental challenge in autonomous driving. While agent-centric models offer high data efficiency, they often struggle to capture the global context necessary for consistent joint forecasts. Conversely, scene-centric models capture global interactions but suffer from data inefficiency. This paper introduces SceneMotion, a novel attention-based framework that bridges this gap. By transforming local agent-centric embeddings into a unified scene-wide latent space using Global Reference Tokens, SceneMotion achieves state-of-the-art performance on the Waymo Interaction Prediction Challenge. Notably, the architecture enables training speeds three times faster than leading competitors while maintaining real-time inference capabilities.
Introduction
In the domain of autonomous driving, motion forecasting has traditionally been divided into two categories: marginal and joint prediction. Marginal models forecast each agent's path independently, which frequently results in "ghost collisions"—scenarios where predicted trajectories for different agents overlap in a way that is physically impossible or socially unrealistic. Joint forecasting aims to solve this by predicting the entire scene simultaneously, but it faces a significant architectural trade-off.
Scene-centric models, which look at the environment as a single entity, are often data-inefficient because they generate only one training sample per scene. Agent-centric models are more efficient as they share weights across many agents, but they often lack the "big picture" context of the road network. SceneMotion is designed to provide the best of both worlds: the data efficiency of agent-centric learning combined with the interaction-awareness of scene-wide forecasting.
Methodology
The SceneMotion Architecture
SceneMotion utilizes a "stack and rearrange" pipeline to process environmental data and agent interactions through several distinct stages:
- Local Agent-Centric Views: The model initially processes small, local "patches" centered on focal agents. Each patch includes the 128 nearest map polylines and 48 neighboring agents, ensuring the model captures immediate surroundings with high granularity.
- Road Environment Descriptor (RED) Tokens: To manage variable-sized local maps, the framework employs a learned reduction mechanism. Cross-attention with RED tokens compresses these local views into fixed-size embeddings ($E_i$), standardizing the input for the subsequent global processing stage.
- The Latent Context Module: This module represents the core innovation of SceneMotion. The model takes the local embeddings and attaches "Global Reference Tokens," which encode the necessary rotation and translation to move from local to global coordinates. These are processed through six blocks of self-attention, creating a scene-wide latent space where agents can effectively "negotiate" their future paths relative to one another.
Interaction Analysis via Clustering
To quantify how well the model resolves interactions, the authors introduced a spatial-temporal analysis method using DBSCAN (Density-Based Spatial Clustering of Applications with Noise). By clustering future waypoints in space and time across different predicted "modes" (possible future outcomes), the authors can measure the model's ability to select non-conflicting trajectories. This allows for a deeper understanding of whether the model is truly learning social navigation or simply averaging potential paths.
Experiments and Results
Dataset and Evaluation
The model was evaluated using the Waymo Open Motion Dataset (WOMD), specifically the Interaction Prediction Challenge. This dataset contains over 1.1 million real-world data points, curated to highlight complex multi-agent interactions in diverse urban and suburban environments.
Performance Metrics
SceneMotion demonstrated superior performance across several key metrics:
- Accuracy: The model achieved a mean Average Precision (mAP) of 0.1789 on the test set, outperforming existing real-time capable models such as GameFormer and Scene Transformer.
- Efficiency: SceneMotion requires only 33 hours of training on 4x A6000 GPUs, compared to 104 hours for the HPTR model. This 3x speedup is attributed to the use of local views, which allow for larger batch sizes and more efficient memory utilization.
- Latency: The model maintains an inference latency of approximately 60ms for 8 focal agents, making it highly suitable for real-world deployment on autonomous vehicle hardware.
Conflict Resolution
The clustering analysis revealed that while nearly 30% of agents had overlapping potential paths when considering all possible modes, this conflict rate dropped to just 6.55% in the model’s "Top-1" (most likely) joint prediction. This indicates that SceneMotion is not merely predicting individual paths but is actively selecting a consistent, non-conflicting version of the future scene.
Discussion and Future Work
SceneMotion represents a significant step forward in balancing efficiency and interaction-awareness. However, the authors identify several areas for future improvement. Currently, the model treats all agents with a standard set of parameters. Future iterations could incorporate mixture models tailored to specific agent types (e.g., distinct logic for cyclists versus heavy trucks) or specific environmental contexts (e.g., high-speed highways versus dense urban intersections).
Furthermore, the authors aim to integrate the waypoint clustering and interaction metrics directly into motion planners. By providing a quantifiable measure of risk and interaction uncertainty, SceneMotion can help autonomous systems make safer, more informed decisions in complex traffic scenarios.
Summary updated