Three-Dimensional Trajectory Prediction with 3DMoTraj Dataset

Hao Zhou, Xu Yang, Mingyu Fan, Lu Qi, Xiangtai Li, Ming-Hsuan Yang, Fei LuoICML 2025 · OpenReview

With the growing interest in embodied and spatial intelligence, accurately predicting trajectories in 3D environments has become increasingly critical. While trajectory prediction for pedestrians and cars in 2D is a well-studied field, 3D trajectory prediction—essential for drones and underwater robots—has significantly lagged behind.

To address this gap, researchers have introduced 3DMoTraj, the first large-scale dataset specifically designed for 3D trajectory prediction featuring unmanned underwater vehicles (UUVs). Alongside the dataset, they propose a novel "divide-and-conquer" method to handle the mathematical complexity of 3D space, achieving state-of-the-art results.

The Challenge of 3D Trajectory Prediction

The authors identify two primary hurdles that have historically bottlenecked 3D trajectory prediction:

  1. Data Scarcity: Prior to this work, there were no dedicated, publicly available datasets specifically tailored for 3D trajectory prediction.
  2. Complexity Explosion (The "9-Parameter" Problem): Mathematically, predicting a 2D position using a Gaussian distribution involves optimizing 5 parameters. Moving to 3D requires 9 parameters. This nearly twofold increase in complexity—driven by inter-axis correlations ($\rho_{xy}, \rho_{yz}, \rho_{xz}$)—creates a much higher-dimensional optimization space. This complexity makes it significantly harder for standard models to converge on accurate predictions, especially in turbulent environments like the open ocean.
Comparison of 2D, 3D, and Decoupled 3D trajectory prediction complexity.

Proposed Methodology: Divide and Conquer

To manage this complexity explosion, the authors propose a two-stage framework that breaks down the high-dimensional uncertainty:

  • Decoupled Trajectory Prediction: Instead of attempting to predict the full 3D correlation at once, the model uses three independent decoders to generate "coarse" predictions for the X, Y, and Z axes separately. By treating axes as independent first, the model establishes a strong spatial baseline and simplifies the initial learning task.
  • Correlated Trajectory Refinement (SCA-LSTM): To correct the errors introduced by ignoring axis correlations in the first step, the authors developed a novel State-Correlation and Aggregation LSTM (SCA-LSTM).
  • A Center LSTM maintains a global summary of the trajectory.
  • Node LSTMs handle individual axes.
  • The State Correlation (SC) module uses a gated mechanism to update axis-specific features based on the global center.
  • The State Aggregation (SA) module uses an attention mechanism to pull the most relevant features from the axes back into the global center.

The 3DMoTraj Dataset

To train and validate their model, the authors collected 3DMoTraj, a new dataset featuring UUVs operating in eight oceanic scenarios.

Sample 3D trajectories from the 3DMoTraj dataset.

The dataset contains over 13,000 frames and includes unique "intention" annotations that act as anchors for the model:

  • Static Intention: Identifies which of the eight "octants" (3D quadrants) the agent is heading toward. By labeling the final goal, the model learns to prioritize the long-term destination over immediate frame-to-frame fluctuations caused by ocean currents.
  • Motion Intention: Indicates whether the agent is speeding up, slowing down, or maintaining a constant velocity.

Key Results and Insights

Extensive experiments demonstrate the effectiveness of the proposed approach:

  • Superior Accuracy: On the 3DMoTraj dataset, the proposed method outperformed the previous best model (S-Implicit) by 14.6% in Average Displacement Error (ADE) and 16.3% in Final Displacement Error (FDE).
  • Efficiency: Despite the two-stage process, the model runs at over 12 FPS on a standard mid-range GPU (NVIDIA 2080 Ti), making it highly viable for real-time robotics applications.
  • 2D Generalization: When tested on standard 2D pedestrian benchmarks (ETH/UCY & SDD), the "decoupled" strategy still improved performance by roughly 4.5% to 5.0%. This suggests that axis-decoupling is a robust strategy even in lower-dimensional tasks.

Limitations and Future Work

While highly effective, the authors note a few limitations. The model occasionally struggles with "failure cases" involving trajectories that have multiple sharp, sudden turns within a short window. Additionally, the current model uses a relatively simple strategy for modeling agent interactions.

Future work will explore integrating 3D point cloud maps of the environment to provide structural constraints near obstacles. The team also aims to collect similar data for Unmanned Aerial Vehicles (UAVs) to evaluate how the model adapts to the distinct dynamics of flight compared to underwater movement.

Summary updated

Abstract

With the growing interest in embodied and spatial intelligence, accurately predicting trajectories in 3D environments has become increasingly critical. However, no datasets have been explicitly designed to study 3D trajectory prediction. To this end, we contribute a 3D motion trajectory (3DMoTraj) dataset collected from unmanned underwater vehicles (UUVs) operating in oceanic environments. Mathematically, trajectory prediction becomes significantly more complex when transitioning from 2D to 3D. To tackle this challenge, we analyze the prediction complexity of 3D trajectories and propose a new method consisting of two key components: decoupled trajectory prediction and correlated trajectory refinement. The former decouples inter-axis correlations, thereby reducing prediction complexity and generating coarse predictions. The latter refines the coarse predictions by modeling their inter-axis correlations. Extensive experiments show that our method significantly improves 3D trajectory prediction accuracy and outperforms state-of-the-art methods. Both the 3DMoTraj dataset and the method are available at https://github.com/zhouhao94/3DMoTraj.
BibTeX
@inproceedings{zhou2025three,
  author = {Hao Zhou and Xu Yang and Mingyu Fan and Lu Qi and Xiangtai Li and Ming-Hsuan Yang and Fei Luo},
  title = {{Three-Dimensional Trajectory Prediction with 3DMoTraj Dataset}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=jkVH7nLzUR},
}
Figure