MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes

Xiaqiang Tang, Weigao Sun, Siyuan Hu, Yiyang Sun, Yafeng Guo

TL;DR

This paper introduces MS-Net, a novel multi-path sparse model for motion prediction in autonomous driving that dynamically adapts its architecture to different traffic scenarios. By using an evolutionary algorithm to learn scenario-specific sub-models from a shared knowledge pool, MS-Net achieves superior prediction accuracy with significantly fewer parameters and reduced inference time compared to unified models.

What Problem Does This Paper Address?

Predicting the future trajectories of dynamic agents like pedestrians and vehicles is crucial for safe autonomous driving. Human behavior is inherently multi-modal and stochastic, making this a challenging task. Existing deep learning approaches often use a single, unified model for all traffic scenarios (e.g., merging, roundabouts, intersections). However, this "one-size-fits-all" approach can lead to sub-optimal performance for individual scenes and inefficient use of computational resources. The paper aims to bridge the gap between diverse traffic scenarios and the design of deep learning models by proposing a method that can learn scenario-specific models while still sharing common knowledge.

Methods

MS-Net employs a multi-path sparse model trained through an evolutionary process. The core idea is to create specialized "sub-models" for different traffic scenarios, rather than relying on a single monolithic model.

Here's a breakdown of the key techniques:

  • Meta-Model and Knowledge Pool: The process starts with a "meta-model" (a pre-trained unified motion forecasting network) which initializes a "Knowledge Pool." This pool acts as a repository of shared knowledge.
  • Evolutionary Algorithm: For each new traffic scenario, an evolutionary algorithm is used to generate and refine sub-models. This involves:
  • Model Evolution: Dynamically adding or removing layers from the meta-model to adapt its complexity to the scenario.
  • Knowledge Transfer: Copying and fine-tuning selected layers from the parent model (or other models in the Knowledge Pool) into the sub-model. This allows for efficient sharing of learned representations while preventing catastrophic forgetting.
  • Scoring Function: A crucial component that balances prediction accuracy with the number of new parameters introduced. This encourages parsimony and prevents overfitting.
  • Hyperparameter Tuning: A random walk algorithm is used to fine-tune hyperparameters for each sub-model, further optimizing its performance for the specific scenario.
  • Sparse Inference: During inference, MS-Net activates only a small subset of parameters relevant to the specific traffic scene, resulting in a scenario-distinct, efficient model.

Datasets

The paper evaluates MS-Net on established pedestrian and vehicle trajectory prediction datasets:

  • ETH and UCY: These datasets contain pedestrian trajectories from urban environments (university campuses, train stations, malls). The task is to predict 12 future timesteps from 8 past timesteps. The authors note they used no semantic or visual information for these datasets, aligning with baseline settings.
  • INTERACTION: This dataset focuses on vehicle trajectories in more complex scenarios like intersections, merging, and roundabouts. The prediction task is to forecast the next 3 seconds based on 1 second of past observations at 10 Hz.

Key Insights and Ideas

  • Scenario-Specific Models Outperform Unified Models: The fundamental insight is that tailoring a model's architecture and parameters to the specific characteristics of a traffic scene leads to better prediction accuracy and efficiency.
  • Evolutionary Approach for Model Design: Using evolutionary algorithms to search for optimal sparse model architectures is a powerful way to automate the design process and discover efficient, scenario-specific solutions.
  • Balancing Accuracy and Parameter Count: The scoring function, which penalizes additional parameters, is key to achieving parameter efficiency and preventing overfitting. This "Occam's Razor" principle helps select the simplest effective model.
  • Knowledge Sharing is Crucial: The Knowledge Pool and Knowledge Transfer mechanisms enable effective sharing of common learned representations across different scenarios, reducing the need to train each scenario from scratch.

Key Results

MS-Net demonstrates significant improvements across various metrics and scenarios:

  • ETH/UCY Datasets: When using the Transformer [33] as a meta-model, MS-Net achieved a 9% reduction in Average Displacement Error (ADE) and a 13.4% reduction in Final Displacement Error (FDE) compared to the original Transformer. It also used one-third of the parameters, reduced inference time by 25%, and used 35% less GPU memory.
  • INTERACTION Dataset:
    • MS-Net ranked 2nd place on the INTERACTION challenge leaderboard.
  • When using AutoBot [17] as a meta-model, MS-Net achieved 20.8% faster inference and used 27.3% less GPU memory compared to AutoBot.
  • Qualitative results showed MS-Net handling challenging prediction instances better than unified models.
  • Efficiency: The paper highlights substantial reductions in computational cost during inference (e.g., 33% and 64% reduction on ETH/UCY and INTERACTION respectively).

Limitations and Future Work

  • Training Time: The paper acknowledges that MS-Net's training time is higher than some baseline models (e.g., 1.3x that of TF, 3x that of AutoBot) due to the sampling of multiple child models.
  • Future Work:
    • Integrating reinforcement learning to enhance evaluation processes for even better prediction precision.
  • Applying MS-Net across various domains to facilitate knowledge transfer between different fields.
  • Exploring the integration of directed acyclic graphs (DAGs) for modeling agent interactions, similar to methods like FJMP [42], which showed strong performance on the INTERACTION dataset.

Summary updated

Abstract

The multi-modality and stochastic characteristics of human behavior make motion prediction a highly challenging task, which is critical for autonomous driving. While deep learning approaches have demonstrated their great potential in this area, it still remains unsolved to establish a connection between multiple driving scenes (e.g., merging, roundabout, intersection) and the design of deep learning models. Current learning-based methods typically use one unified model to predict trajectories in different scenarios, which may result in sub-optimal results for one individual scene. To address this issue, we propose Multi-Scenes Network (aka. MS-Net), which is a multi-path sparse model trained by an evolutionary process. MS-Net selectively activates a subset of its parameters during the inference stage to produce prediction results for each scene. In the training stage, the motion prediction task under differentiated scenes is abstracted as a multi-task learning problem, an evolutionary algorithm is designed to encourage the network search of the optimal parameters for each scene while sharing common knowledge between different scenes. Our experiment results show that with substantially reduced parameters, MS-Net outperforms existing state-of-the-art methods on well-established pedestrian motion prediction datasets, e.g., ETH and UCY, and ranks the 2nd place on the INTERACTION challenge.
BibTeX
@article{tang2024ms,
  author = {Xiaqiang Tang and Weigao Sun and Siyuan Hu and Yiyang Sun and Yafeng Guo},
  title = {{MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
}
Figure