Abstract
Large Language Models (LLMs) have demonstrated remarkable zero-shot reasoning and extrapolation capabilities, yet their application to safety-critical tasks like vehicle trajectory prediction remains challenging. This work introduces a novel framework that utilizes frozen LLMs by "reprogramming" raw sensor data into semantically rich "pseudo-language" tokens. By bypassing traditional text-based coordinate representations, the system avoids token bloat and semantic sparsity. Evaluated on the nuScenes dataset, the proposed method achieves performance comparable to specialized autonomous driving models while operating at real-time speeds (21Hz), proving that LLMs can effectively serve as trajectory predictors when their internal logic is properly engaged.
Introduction
Predicting the future paths of vehicles is a fundamental requirement for autonomous driving systems. While LLMs are natural candidates for extrapolation tasks, three primary obstacles hinder their direct application: the "Fake LLM" trap (where surrounding encoders do the actual work), token bloat resulting from numerical text conversion, and the semantic sparsity of raw numbers compared to natural language.
Methodology
The proposed framework ensures that the LLM's internal weights are the primary drivers of prediction. The architecture consists of three core modules designed to bridge the gap between sensor data and linguistic embeddings:
- Traffic Scene Encoder: This module uses cross-attention mechanisms to capture the spatial relationships between the target vehicle and its neighbors at every timestamp.
- Reprogramming Adapter: This component maps extracted scene features into the LLM’s existing text embedding space. It represents traffic features as combinations of "text prototypes," effectively translating sensor data into a format the LLM can process as if it were reading a sentence.
- Linear Trajectory Decoder: To ensure the LLM performs the actual prediction work, a simple linear layer is used to convert the LLM's output back into future coordinates, preventing complex decoders from "cheating."
Experimental Results
The model was rigorously evaluated on the nuScenes dataset, which contains 1,000 real-world driving scenes featuring complex multi-agent interactions. Using a 7B-8B parameter backbone, the framework achieved an Average Displacement Error (ADE) of 0.714m and a Final Displacement Error (FDE) of 6.062m over a 6-second horizon.
As shown in the evaluation metrics, the framework significantly outperforms zero-shot (ZS-LLM) and chain-of-thought (CoT-LLM) prompting methods. Notably, it reaches performance levels comparable to specialized, non-LLM models like Trajectron++ and AgentFormer. Furthermore, the system operates at 21Hz, making it suitable for real-time deployment.
Key Insights
A critical finding of this study was the "Identity Mapping" test: replacing the LLM with a simple pass-through layer caused a massive spike in error, confirming that the frozen LLM's internal logic is essential to the prediction process. Additionally, the "reprogrammed" tokens proved robust; shuffling input sequences resulted in significantly less performance degradation compared to raw numerical baselines, suggesting the adapter successfully imbues tokens with local semantic meaning.
Conclusion and Future Work
This research demonstrates that off-the-shelf LLMs possess an intrinsic ability to predict trajectories when sensor data is correctly aligned with their embedding space. While the model is competitive, current limitations include the absence of High-Definition (HD) map data (e.g., lanes and traffic lights). Future work will focus on incorporating these environmental features and refining the model's understanding of complex multi-agent interactions to further bridge the gap with state-of-the-art specialized predictors.
Summary updated