Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models

Zhengxing Lan, Hongbo Li, Lingshan Liu, Bo Fan, Yisheng Lv, Yilong Ren, Zhiyong CuiIEEE Trans. Intell. Veh 2024 · DOI · OpenReview

TL;DR

This paper introduces Traj-LLM, a novel framework for autonomous vehicle trajectory prediction that leverages the power of pre-trained Large Language Models (LLMs) without explicit prompt engineering. By integrating LLMs with a new lane-aware probabilistic learning module and a multi-modal Laplace decoder, Traj-LLM achieves state-of-the-art performance, even in few-shot learning scenarios, demonstrating a more universal and adaptable approach to predicting agent motion.

What Problem Does This Paper Address?

Predicting the future movements of vehicles is crucial for safe and efficient autonomous driving. While existing methods have made significant progress, they often struggle with understanding the complex "semantics" of a driving scene – the nuanced interactions between vehicles, road infrastructure, and the overall context. This lack of deep scene comprehension limits their ability to predict trajectories as accurately and intuitively as human drivers. Current approaches often rely on carefully crafted prompts to guide LLMs, which can be cumbersome and introduce biases. This paper aims to bridge this gap by directly integrating LLMs into trajectory prediction, harnessing their inherent reasoning and comprehension abilities without the need for manual prompt design.

Methods

Traj-LLM is built upon four key components:

  • Sparse Context Joint Coding: This module transforms raw agent and scene data (like historical trajectories and map information) into a format that LLMs can readily understand. It uses embedding networks and attention mechanisms to capture local dependencies.
  • High-level Interaction Modeling: Here, pre-trained LLMs are employed to model complex, high-level interactions within the driving scene. To make this efficient, the authors use Parameter-Efficient Fine-Tuning (PEFT) with the Low-Rank Adaptation (LoRA) technique. LoRA injects trainable parameters into the LLM's attention layers, allowing it to adapt to trajectory prediction tasks without retraining the entire model.
  • Lane-aware Probabilistic Learning: This novel module mimics how skilled human drivers focus on relevant lane segments. It uses the Mamba module, a selective state space model, to continuously estimate the likelihood of an agent's motion aligning with potential lanes. This enhances scene understanding and guides predictions towards plausible lane maneuvers.
  • Multi-modal Laplace Decoder: Since future trajectories can be uncertain and have multiple possibilities (multi-modal), this decoder generates a set of diverse predictions. It uses a mixture model where each component follows a Laplace distribution, parameterized by location and uncertainty, to capture this multi-modality.

Datasets

The primary dataset used for training and evaluation is nuScenes. This dataset is a widely used benchmark for autonomous driving, featuring 1,000 driving scenarios collected from various cities. It includes multimodal sensor data and is known for its comprehensive depiction of urban traffic dynamics. The paper uses 2-second historical sequences to predict 6-second future trajectories.

Key Insights and Ideas

  • LLMs without Prompts: The core innovation is demonstrating that LLMs can be effectively used for trajectory prediction without the need for explicit, hand-crafted prompts. This opens up a more general and adaptable paradigm.
  • Mimicking Human Driving: The "lane-aware probabilistic learning" module is a clever way to imbue the model with human-like cognitive functions, specifically the ability to focus on relevant lanes, which significantly improves prediction accuracy.
  • Efficiency through PEFT: Using LoRA for fine-tuning LLMs is crucial. It allows the model to leverage the vast knowledge of pre-trained LLMs without the prohibitive computational cost of full fine-tuning, making the approach more practical.
  • Few-Shot Prowess: Traj-LLM's strong performance even with significantly reduced training data (e.g., 50%) highlights the power of LLMs' pre-trained knowledge and the model's robust generalization capabilities.

Key Results

  • State-of-the-Art Performance: Traj-LLM consistently outperforms existing state-of-the-art methods across various metrics (minADE, minFDE, MR) for both K=5 and K=10 prediction modalities.
  • Significant Improvements: For instance, Traj-LLM shows substantial enhancements over methods like MUSE-VAE and LAformer, particularly in reducing prediction errors (minFDE) and miss rates (MR).
  • Few-Shot Advantage: With only 50% of the training data, Traj-LLM outperforms many baselines that rely on 100% of the data, demonstrating remarkable generalization.
  • Computational Efficiency: Despite its advanced capabilities, Traj-LLM achieves competitive inference speeds, outperforming some models with fewer trainable parameters.

Limitations and Future Work

  • LLM Size: While PEFT makes fine-tuning efficient, the underlying pre-trained LLMs are still large, which can impact deployment on resource-constrained edge devices.
  • Generalization to Diverse Scenarios: While the paper shows strong performance on nuScenes, further validation on a wider range of driving environments and scenarios would be beneficial.
  • Future Work: The authors suggest exploring further integration of LLMs for more complex tasks in autonomous driving and investigating more efficient LLM architectures for real-time deployment. The success of the lane-aware module also hints at potential for incorporating other domain-specific knowledge.

Summary updated

Abstract

Predicting the future trajectories of dynamic traffic actors is a cornerstone task in autonomous driving. Though existing notable efforts have resulted in impressive performance improvements, a gap persists in scene cognitive and understanding of the complex traffic semantics. This paper proposes Traj-LLM, the first to investigate the potential of using Large Language Models (LLMs) without explicit prompt engineering to generate future motion from agents' past/observed trajectories and scene semantics. Traj-LLM starts with sparse context joint coding to dissect the agent and scene features into a form that LLMs understand. On this basis, we innovatively explore LLMs' powerful comprehension abilities to capture a spectrum of high-level scene knowledge and interactive information. Emulating the human-like lane focus cognitive function and enhancing Traj-LLM's scene comprehension, we introduce lane-aware probabilistic learning powered by the pioneering Mamba module. Finally, a multi-modal Laplace decoder is designed to achieve scene-compliant multi-modal predictions. Extensive experiments manifest that Traj-LLM, fortified by LLMs' strong prior knowledge and understanding prowess, together with lane-aware probability learning, outstrips state-of-the-art methods across evaluation metrics. Moreover, the few-shot analysis further substantiates Traj-LLM's performance, wherein with just 50% of the dataset, it outperforms the majority of benchmarks relying on complete data utilization. This study explores equipping the trajectory prediction task with advanced capabilities inherent in LLMs, furnishing a more universal and adaptable solution for forecasting agent motion in a new way.
BibTeX
@article{lan2024traj,
  author = {Zhengxing Lan and Hongbo Li and Lingshan Liu and Bo Fan and Yisheng Lv and Yilong Ren and Zhiyong Cui},
  title = {{Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models}},
  journal = {IEEE Trans. Intell. Veh.},
  year = {2024},
  doi = {10.1109/tiv.2024.3418522},
}
Figure