Layout Sequence Prediction From Noisy Mobile Modality

Haichao Zhang, Yi Xu, Hongsheng Lu, Takayuki Shimizu, Yun Fu

Abstract

Trajectory prediction is a cornerstone of autonomous navigation and robotics, yet traditional models struggle with the "line-of-sight" limitation of visual sensors. Real-world environments are fraught with obstructions, leading to incomplete or noisy trajectory data. To address this, we introduce LTrajDiff, a novel framework that treats occluded or out-of-sight objects with the same priority as visible ones. By fusing visual data with noisy mobile sensor data (IMU, Wi-Fi) from handheld devices, LTrajDiff employs a denoising diffusion model to predict precise 3D layout sequences. Utilizing a coarse-to-fine diffusion strategy and specialized modules like the Siamese Masked Encoding and Modality Fusion Module (MFM), our approach implicitly infers object size and projection status even from significantly obstructed sequences. LTrajDiff achieves state-of-the-art (SOTA) results on the Vi-Fi and H3D datasets, particularly in extreme scenarios with minimal visual input, marking a significant step forward in multi-modal trajectory prediction.

Introduction

Standard trajectory prediction for autonomous driving and robotics relies heavily on computer vision. While cameras provide high-resolution spatial data, they suffer from a fundamental flaw: the requirement for a clear line-of-sight. If a pedestrian walks behind a vehicle or turns a corner, the camera loses them, creating dangerous "blind spots" and fragmented trajectories.

Conversely, mobile devices (smartphones, wearables) emit persistent signals—such as Inertial Measurement Unit (IMU) data and Wi-Fi—that are not blocked by physical obstacles. However, this "mobile modality" is notoriously noisy and lacks "layout" information; a phone can indicate movement, but it cannot inherently determine the 3D bounding box or the physical dimensions of the person carrying it within a camera's coordinate system. LTrajDiff bridges this gap by combining the precise-but-occludable visual modality with the persistent-but-noisy mobile modality.

The LTrajDiff Framework

The core of our approach is built on Denoising Diffusion Probabilistic Models (DDPM). Rather than attempting to regress a trajectory directly, which often leads to drift during long occlusions, LTrajDiff learns to "denoise" a sequence of layout points. This process is guided by both mobile and visual cues through a coarse-to-fine strategy.

Key Architectural Innovations

  1. Random Mask Strategy (RMS): To ensure the model does not become over-reliant on visual data, we randomly mask chunks of visual information during training. This forces the network to learn how to leverage mobile signals to "fill in the blanks" when the camera view is obstructed.
  2. Siamese Masked Encoding Module: This module consists of two specialized components:
  3. Temporal Alignment Module (TAM): This synchronizes high-frequency, noisy mobile signals with the lower-frequency visual frames.
  4. Layout Extracting Module (LEM): Acting as the "spatial brain," the LEM infers an object's physical dimensions (size and depth) from the available visual frames and applies this knowledge to segments of the trajectory where the object is invisible.
  5. Modality Fusion Module (MFM): A cross-modal gating mechanism that merges features from both sensors into a unified representation, which then informs the diffusion decoder.

Experimental Evaluation

We evaluated LTrajDiff on two primary multi-modal benchmarks: the Vi-Fi Multi-modal Dataset (synchronized video and smartphone sensor data) and the H3D Dataset (large-scale 3D multi-object tracking).

Performance in Extreme Scenarios

LTrajDiff significantly outperformed state-of-the-art baselines, including LSTMs, Transformers, and UNets, particularly when visual data was scarce:

  • Random Obstructions: On the Vi-Fi dataset, LTrajDiff achieved an IoU-D (Intersection over Union with Depth) of 0.69, a substantial improvement over the 0.43 achieved by the next best baseline (UNet).
  • Extremely Short Inputs: In stress tests where the model was provided with only 1/50th of the visual sequence (effectively a single frame), it successfully reconstructed plausible 3D trajectories. In contrast, traditional models like ViTag and standard LSTMs failed to converge.
  • Quantitative Accuracy: On the H3D dataset, the model reduced the Mean Square Error per Timestamp (MSE-T) to 2.72, nearly halving the error rate of standard Transformer-based predictors (5.17).

Discussion and Insights

The success of LTrajDiff stems from two conceptual shifts. First, we treat mobile signals as a global anchor. Even when noisy, the mobile signal provides a continuous temporal thread that prevents the model from losing the object during long-term occlusions. Second, we use vision as a spatial template. The model uses "In-Sight" moments to learn the object's layout (bounding box size) and then projects that template onto the noisy mobile path during "Out-of-Sight" moments.

Furthermore, the generative nature of diffusion models makes them uniquely suited for this task. Unlike traditional Seq2Seq models that struggle with "drift" during long gaps, diffusion models maintain structural consistency—such as stable bounding box sizes—while hallucinating the missing portions of a path.

Conclusion and Future Work

LTrajDiff is the first work to address severely obstructed and extremely short layout sequences by combining vision with noisy mobile modalities. While the model sets a new benchmark for accuracy, the iterative nature of diffusion models introduces inference latency challenges. Future research will focus on model pruning and sampling acceleration techniques to enable real-time deployment in autonomous vehicles. Additionally, moving beyond bounding boxes toward full 3D mesh representations could further enhance scene understanding in complex urban environments.

Summary updated

Abstract

Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. Nevertheless, real-world situations often involve obstructed cameras, missed objects, or objects out of sight due to environmental factors, leading to incomplete or noisy trajectories. To overcome these limitations, we propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount out-of-sight constraints, albeit introducing new challenges such as modality fusion, noisy data, and the absence of spatial layout and object size information. We employ a denoising diffusion model to predict precise layout sequences from noisy mobile data using a coarse-to-fine diffusion strategy, incorporating the RMS, Siamese Masked Encoding Module, and MFM. Our model predicts layout sequences by implicitly inferring object size and projection status from a single reference timestamp or significantly obstructed sequences. Achieving SOTA results in randomly obstructed experiments and extremely short input experiments, our model illustrates the effectiveness of leveraging noisy mobile data. In summary, our approach offers a promising solution to the challenges faced by layout sequence and trajectory prediction models in real-world settings, paving the way for utilizing sensor data from mobile phones to accurately predict pedestrian bounding box trajectories. To the best of our knowledge, this is the first work that addresses severely obstructed and extremely short layout sequences by combining vision with noisy mobile modality, making it the pioneering work in the field of layout sequence trajectory prediction.
BibTeX
@inproceedings{zhang2023layout,
  author = {Haichao Zhang and Yi Xu and Hongsheng Lu and Takayuki Shimizu and Yun Fu},
  title = {{Layout Sequence Prediction From Noisy Mobile Modality}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611936},
}
Figure