Multi-Task Trajectory Prediction Using a Vehicle-Lane Disentangled Conditional Variational Autoencoder

Haoyang Chen, Na Li, Hangguan Shan, Eryun Liu, Zhiyu XiangSensors 2025 · DOI

TL;DR

This paper introduces MS-SLV, a novel generative model for autonomous driving that better predicts future vehicle trajectories by jointly considering vehicle motion and static map information. It achieves this by disentangling vehicle-specific intentions from scene context using a structured latent space and an auxiliary lane prediction task, leading to more accurate and diverse predictions.

What Problem Does This Paper Address?

Accurate trajectory prediction is crucial for autonomous driving, requiring the integration of dynamic vehicle data (like position and velocity) with static high-definition (HD) map information. However, existing methods often struggle to effectively capture how the environment's spatial context changes as a vehicle moves. They also tend to use unstructured latent spaces, which limits their ability to generate diverse yet contextually coherent predictions. This can lead to either overly deterministic forecasts or predictions that don't make sense within the driving scene. The paper aims to bridge this gap by creating a unified framework that aligns evolving scene semantics with agent movement and models uncertainty in a structured way.

Methods

The core of MS-SLV is a Multi-Source Input with Structured Latent Variable Model (MS-SLV). Key innovations include:

  • Time-Aware Scene Encoder: This component aligns HD map features with vehicle motion over time, capturing how scene semantics evolve relative to the vehicle's movement. This is a departure from static map encodings.
  • Structured Latent Variable Model: Instead of a single unstructured latent space, MS-SLV uses a structured approach that explicitly disentangles:
  • Vehicle-specific latent variables ($z_{VP}$): Capturing individual agent intent.
  • Lane-specific latent variables ($z_{CP}$): Capturing scene-level constraints from lane information.
  • Shared latent variables ($z_S$): Capturing common factors between vehicle and scene, promoting consistency. This is achieved through semantic partitioning and KL divergence regularization, aiming to prevent redundancy and mode collapse.

  • Multi-Task Generative Framework: The model jointly predicts future trajectories and lane sequences. The auxiliary lane sequence prediction task acts as a form of targeted supervision, improving scene understanding and the learning of latent variables. This multi-task approach reinforces prediction robustness.

The framework uses a Conditional Variational Autoencoder (CVAE) backbone, with a modified Evidence Lower Bound (ELBO) that explicitly accounts for the structured latent space. Regularization techniques like Total Correlation (TC) and Dimension-Wise KL divergence (DW-KL) are employed to encourage disentanglement and stability.

Datasets

The primary dataset used for training and evaluation is nuScenes. This dataset provides high-definition maps and annotated 3D bounding boxes for 1000 driving scenes, offering rich multimodal data for autonomous driving research.

Key Insights and Ideas

  • Disentangling Intent from Context is Key: The paper strongly advocates for separating vehicle-specific intentions ($z_{VP}$) from scene-level constraints ($z_{CP}$ and $z_S$). This structured approach leads to more interpretable and accurate predictions, especially in complex scenarios.
  • Lane Prediction as Supervision: Using lane sequence prediction as an auxiliary task significantly improves the model's ability to learn structured latent variables and enhances overall prediction accuracy and diversity. It acts as a form of implicit regularization.
  • Time-Aware Scene Encoding Matters: Static map encodings are insufficient. Capturing how lane geometry and road layout evolve relative to vehicle motion is crucial for understanding dynamic driving scenarios.
  • Balancing Accuracy and Diversity: The structured latent space and multi-task learning framework effectively address the common trade-off between generating accurate predictions and diverse, plausible futures.

Key Results

  • Accuracy Improvements: MS-SLV achieved significant reductions in prediction errors on the nuScenes dataset:
  • 12.37% reduction in Average Displacement Error (ADE).
  • 7.67% reduction in Final Displacement Error (FDE).
  • Diversity and Safety Enhancements:
    • 26% reduction in top-5 Miss Rate (MR5).
  • 33% reduction in top-10 Miss Rate (MR10).
  • 3% reduction in Off-Road Rate (ORR).
  • Efficiency: The model demonstrates real-time performance with an average inference time of 40.6 ms per sample, making it suitable for deployment. It also shows improved training efficiency compared to a baseline.
  • Qualitative Results: Visualizations show MS-SLV generating accurate and temporally consistent trajectories in simple scenarios, and multiple plausible, scene-consistent futures in complex scenarios like intersections and merges.

Limitations and Future Work

  • Computational Complexity: The framework involves relatively high computational complexity due to latent variable sampling and lane sequence prediction.
  • HD Map Dependency: The model relies heavily on accurate HD map data, which may not always be available or up-to-date in real-world scenarios.
  • Generalization: While evaluated on nuScenes, further assessment on diverse datasets is needed to verify generalization capabilities.
  • Future Work Suggestions:
    • Incorporate environmental factors like weather conditions (rain, snow, fog) and road surface states (wet, icy).
  • Explore broader datasets and simulation environments for more robust testing.
  • Further reduce prediction latency for real-time deployment.

Summary updated

Abstract

Trajectory prediction under multimodal information is critical for autonomous driving, necessitating the integration of dynamic vehicle states and static high-definition (HD) maps to model complex agent-scene interactions effectively. However, existing methods often employ static scene encodings and unstructured latent spaces, limiting their ability to capture evolving spatial contexts and produce diverse yet contextually coherent predictions. To tackle these challenges, we propose <i>MS-SLV</i>, a novel generative framework that introduces (1) a time-aware scene encoder that aligns HD map features with vehicle motion to capture evolving scene semantics and (2) a structured latent model that explicitly disentangles agent-specific intent and scene-level constraints. Additionally, we introduce an auxiliary lane prediction task to provide targeted supervision for scene understanding and improve latent variable learning. Our approach jointly predicts future trajectories and lane sequences, enabling more interpretable and scene-consistent forecasts. Extensive evaluations on the nuScenes dataset demonstrate the effectiveness of MS-SLV, achieving a 12.37% reduction in average displacement error and a 7.67% reduction in final displacement error over state-of-the-art methods. Moreover, MS-SLV significantly improves multi-modal prediction, reducing the top-5 Miss Rate (MR5) and top-10 Miss Rate (MR10) by 26% and 33%, respectively, and lowering the Off-Road Rate (ORR) by 3%, as compared with the strongest baseline in our evaluation.
BibTeX
@inproceedings{chen2025multi,
  author = {Haoyang Chen and Na Li and Hangguan Shan and Eryun Liu and Zhiyu Xiang},
  title = {{Multi-Task Trajectory Prediction Using a Vehicle-Lane Disentangled Conditional Variational Autoencoder}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25144505},
}
Figure