Abstract
Trajectory prediction is a cornerstone of autonomous navigation and robotics, yet traditional models struggle with the "line-of-sight" limitation of visual sensors. Real-world environments are fraught with obstructions, leading to incomplete or noisy trajectory data. To address this, we introduce LTrajDiff, a novel framework that treats occluded or out-of-sight objects with the same priority as visible ones. By fusing visual data with noisy mobile sensor data (IMU, Wi-Fi) from handheld devices, LTrajDiff employs a denoising diffusion model to predict precise 3D layout sequences. Utilizing a coarse-to-fine diffusion strategy and specialized modules like the Siamese Masked Encoding and Modality Fusion Module (MFM), our approach implicitly infers object size and projection status even from significantly obstructed sequences. LTrajDiff achieves state-of-the-art (SOTA) results on the Vi-Fi and H3D datasets, particularly in extreme scenarios with minimal visual input, marking a significant step forward in multi-modal trajectory prediction.
Introduction
Standard trajectory prediction for autonomous driving and robotics relies heavily on computer vision. While cameras provide high-resolution spatial data, they suffer from a fundamental flaw: the requirement for a clear line-of-sight. If a pedestrian walks behind a vehicle or turns a corner, the camera loses them, creating dangerous "blind spots" and fragmented trajectories.
Conversely, mobile devices (smartphones, wearables) emit persistent signals—such as Inertial Measurement Unit (IMU) data and Wi-Fi—that are not blocked by physical obstacles. However, this "mobile modality" is notoriously noisy and lacks "layout" information; a phone can indicate movement, but it cannot inherently determine the 3D bounding box or the physical dimensions of the person carrying it within a camera's coordinate system. LTrajDiff bridges this gap by combining the precise-but-occludable visual modality with the persistent-but-noisy mobile modality.
The LTrajDiff Framework
The core of our approach is built on Denoising Diffusion Probabilistic Models (DDPM). Rather than attempting to regress a trajectory directly, which often leads to drift during long occlusions, LTrajDiff learns to "denoise" a sequence of layout points. This process is guided by both mobile and visual cues through a coarse-to-fine strategy.
Key Architectural Innovations
- Random Mask Strategy (RMS): To ensure the model does not become over-reliant on visual data, we randomly mask chunks of visual information during training. This forces the network to learn how to leverage mobile signals to "fill in the blanks" when the camera view is obstructed.
- Siamese Masked Encoding Module: This module consists of two specialized components:
- Temporal Alignment Module (TAM): This synchronizes high-frequency, noisy mobile signals with the lower-frequency visual frames.
- Layout Extracting Module (LEM): Acting as the "spatial brain," the LEM infers an object's physical dimensions (size and depth) from the available visual frames and applies this knowledge to segments of the trajectory where the object is invisible.
- Modality Fusion Module (MFM): A cross-modal gating mechanism that merges features from both sensors into a unified representation, which then informs the diffusion decoder.
Experimental Evaluation
We evaluated LTrajDiff on two primary multi-modal benchmarks: the Vi-Fi Multi-modal Dataset (synchronized video and smartphone sensor data) and the H3D Dataset (large-scale 3D multi-object tracking).
Performance in Extreme Scenarios
LTrajDiff significantly outperformed state-of-the-art baselines, including LSTMs, Transformers, and UNets, particularly when visual data was scarce:
- Random Obstructions: On the Vi-Fi dataset, LTrajDiff achieved an IoU-D (Intersection over Union with Depth) of 0.69, a substantial improvement over the 0.43 achieved by the next best baseline (UNet).
- Extremely Short Inputs: In stress tests where the model was provided with only 1/50th of the visual sequence (effectively a single frame), it successfully reconstructed plausible 3D trajectories. In contrast, traditional models like ViTag and standard LSTMs failed to converge.
- Quantitative Accuracy: On the H3D dataset, the model reduced the Mean Square Error per Timestamp (MSE-T) to 2.72, nearly halving the error rate of standard Transformer-based predictors (5.17).
Discussion and Insights
The success of LTrajDiff stems from two conceptual shifts. First, we treat mobile signals as a global anchor. Even when noisy, the mobile signal provides a continuous temporal thread that prevents the model from losing the object during long-term occlusions. Second, we use vision as a spatial template. The model uses "In-Sight" moments to learn the object's layout (bounding box size) and then projects that template onto the noisy mobile path during "Out-of-Sight" moments.
Furthermore, the generative nature of diffusion models makes them uniquely suited for this task. Unlike traditional Seq2Seq models that struggle with "drift" during long gaps, diffusion models maintain structural consistency—such as stable bounding box sizes—while hallucinating the missing portions of a path.
Conclusion and Future Work
LTrajDiff is the first work to address severely obstructed and extremely short layout sequences by combining vision with noisy mobile modalities. While the model sets a new benchmark for accuracy, the iterative nature of diffusion models introduces inference latency challenges. Future research will focus on model pruning and sampling acceleration techniques to enable real-time deployment in autonomous vehicles. Additionally, moving beyond bounding boxes toward full 3D mesh representations could further enhance scene understanding in complex urban environments.
Summary updated