Driving is inherently a social negotiation. When a vehicle changes lanes, navigates a crowded intersection, or merges onto a highway, its future path is not merely a continuation of its past velocity vector. Instead, it is a dynamic reaction to the actions and perceived intentions of surrounding road users.
Traditional sequence models, such as standard Long Short-Term Memory (LSTM) networks, excel at tracking an individual agent's temporal history but treat other vehicles as static obstacles. To build safe and socially compliant autonomous systems, motion forecasting must transition from single-agent trajectory extrapolation to interaction-aware prediction.
Mapping the Social Grid with Attention and Graphs
How do we model the mutual influence between multiple moving objects? Early approaches attempted to discretize the space around a target vehicle to capture its immediate surroundings. In Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks, Xiaoyu Mo et al. combined LSTMs with Convolutional Neural Networks (CNNs), using the LSTM to capture individual temporal dynamics and a CNN to process a spatial grid representing surrounding vehicles. While effective in dense traffic, rigid spatial grids struggle with varying numbers of agents and long-range dependencies.
To solve this, researchers turned to attention mechanisms, which dynamically weight the relevance of surrounding agents regardless of grid boundaries. In Relational Recurrent Neural Networks For Vehicle Trajectory Prediction, Kaouther Messaoud et al. introduced Relational Recurrent Neural Networks (RRNNs), pairing temporal LSTMs with an attention layer to capture non-local inter-vehicle interactions on highways. Similarly, in Intelligent Vehicle Moving Trajectory Prediction Based on Residual Attention Network, Zhengcai Yang et al. proposed a residual attention mechanism (RA-LSTM) to adaptively scale the influence of neighboring vehicles on a target vehicle's future path.
Human drivers do not pay equal attention to all vehicles; they selectively track agents that pose an immediate conflict. Mimicking this cognitive behavior, the authors of Attention Based Vehicle Trajectory Prediction designed an attention-based model that replicates human selective attention, allowing the network to focus on critical interactants and generate multiple plausible future paths. Haicheng Liao et al. extended this human-centric philosophy in Human Observation-Inspired Trajectory Prediction for Autonomous Driving in Mixed-Autonomy Traffic Environments with their GaVa framework, which explicitly incorporates principles of human visual attention and cognitive observation to navigate mixed-autonomy environments safely.
As traffic scenarios grow more complex, representing interactions as a graph becomes highly intuitive: vehicles are nodes, and their interactions are edges. Xinshuo Weng et al. leveraged Graph Neural Networks (GNNs) in PTP: Parallelized Tracking and Prediction With Graph Neural Networks and Diversity Sampling, modeling social interactions directly on a graph to prevent error propagation between tracking and forecasting.
To capture both time and space simultaneously, Hongbo Li et al. introduced TPASTGCN in Trajectory Prediction with Attention-Based Spatial–Temporal Graph Convolutional Networks for Autonomous Driving. This architecture allows the model to map vehicle-to-vehicle interactions alongside static traffic constraints, such as traffic lights, within a unified topological structure. Interestingly, the mathematical utility of these spatial-temporal architectures is robust across domains; in Exploitation of Hidden Context in Dynamic Movement Forecasting: A Neural Network Journey from Recurrent to Graph Neural Networks and General Purpose Transformers, Lukas Schelenz et al. demonstrated that similar hybrid CNN-LSTM and transformer principles effectively capture the hidden context and spatial-temporal dynamics of player movements in sports analytics.
Managing Multi-Modality and Physical Feasibility
An interaction-aware model must also confront the inherent uncertainty of human intent. At a fork in the road, a driver might go left or right; a single deterministic line is insufficient. Models must be multi-modal. In Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting, Jean Mercat et al. addressed this by utilizing multi-head attention to output multi-modal joint probability distributions, capturing several distinct, plausible futures for multiple interacting vehicles at once.
However, unconstrained multi-modal outputs can produce physically impossible trajectories, such as a vehicle driving through a building or off the road. To enforce realism, models must anchor predictions to the environment. In ControlMTR: Control-Guided Motion Transformer with Scene-Compliant Intention Points for Feasible Motion Prediction, Jiawei Sun 0006 et al. developed ControlMTR, which generates "scene-compliant" intention points and incorporates control commands directly into a Motion Transformer. This ensures that the predicted trajectories are not just kinematically possible but also strictly adhere to road boundaries.
Another way to guarantee geometric compliance is to shift from object-centric coordinates to grid-based representations. In LEGO-Motion: Learning-Enhanced Grids with Occupancy Instance Modeling for Class-Agnostic Motion Prediction, Kangan Qian et al. introduced LEGO-Motion, a framework that injects instance-level features into Bird's-Eye View (BEV) occupancy grids. By modeling both individual instances and the continuous occupancy grid, the system achieves class-agnostic motion prediction that excels in unpredictable, open-world urban environments.
Fusing High-Level Semantics and Environmental Context
To predict where an agent is going, a model needs to understand where the agent is. This requires fusing raw trajectory data with rich environmental context, such as HD maps and sensor data. In MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR, Yiqian Gan et al. proposed a Multi-Granular Transformer that fuses multi-granular map features with 3D LiDAR data, providing the deep network with a highly detailed geometric and structural understanding of the scene.
Beyond geometry, traffic scenarios are governed by semantic rules and subtle human cues that are difficult to encode via traditional tensors. To capture this high-level context, Xiaoji Zheng et al. introduced Large Language Models (LLMs) into the motion prediction pipeline in Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving. By converting traffic scenes into visual and textual prompts, the authors leverage the semantic reasoning capabilities of LLMs to extract rich contextual insights, which are then integrated back into traditional prediction networks to handle highly intricate traffic situations.
Closing the Loop from Prediction to Planning
Ultimately, predicting the future is only useful if an autonomous vehicle can use that prediction to plan its own safe actions. In highly interactive settings, prediction and planning cannot operate in isolation; the ego vehicle's planned path will actively alter the future trajectories of surrounding agents.
To close this loop, Yuxiao Chen et al. developed Interactive Joint Planning (IJP) in Interactive Joint Planning for Autonomous Vehicles. Instead of treating predictions as static obstacles, the authors treat deep-learning-based trajectory predictions as priors within a Model Predictive Control (MPC) optimization framework. This allows the ego vehicle to jointly optimize its own trajectory alongside the predicted reactions of surrounding agents.
This tight coupling between prediction and planning is equally vital in extreme, high-speed environments like autonomous racing. In FSDP: Fast and Safe Data-Driven Overtaking Trajectory Planning for Head-to-Head Autonomous Racing Competitions, Cheng Hu et al. introduced a data-driven planner for head-to-head racing that utilizes Sparse Gaussian Processes to predict opponent trajectories. These predictions feed directly into a bi-level Quadratic Programming trajectory generator, enabling safe, real-time overtaking maneuvers under intense, dynamic conditions.
Whether through the probabilistic modeling of Dynamic Bayesian Networks for early maneuver detection (A Dynamic Bayesian Network for Vehicle Maneuver Prediction in Highway Driving Scenarios: Framework and Verification by Junxiang Li et al.), the variance reduction of ensemble learning frameworks (An Ensemble Learning Framework for Vehicle Trajectory Prediction in Interactive Scenarios by Zirui Li et al.), or the utilization of brief historical windows for long-term forecasting in public human-robot telecollaboration (Human-Machine Telecollaboration Accelerates the Safe Deployment of Large-Scale Autonomous Robots During the COVID-19 Pandemic by Zhongxu Hu et al.), modern trajectory prediction has firmly shifted. By treating the road not as a collection of isolated moving points, but as a deeply interconnected, context-rich social graph, interaction-aware motion prediction provides the foundational foresight necessary for truly safe autonomous navigation.
Key Foundational Work
Relational Recurrent Neural Networks For Vehicle Trajectory Prediction (2019) — contributed 2 concept elements to this direction.
Vehicle Trajectories from Unlabeled Data Through Iterative Plane Registration (2019) — contributed 1 concept elements to this direction.
Feed-forwards meet recurrent networks in vehicle trajectory prediction (2019) — contributed 1 concept elements to this direction.