Layout Sequence Prediction From Noisy Mobile Modality

Haichao Zhang, Yi Xu, Hongsheng Lu, Takayuki Shimizu, Yun Fu

TL;DR

Researchers have developed LTrajDiff, a denoising diffusion model that predicts 3D pedestrian "layouts" (bounding boxes and depth) by fusing noisy mobile sensor data (like IMU and Wi-Fi) with limited camera views. This approach effectively solves the "out-of-sight" problem, allowing autonomous systems to track individuals even when they are hidden behind walls or cars.

What Problem Does This Paper Address?

Standard trajectory prediction for autonomous driving and robotics relies heavily on computer vision. While cameras provide high-resolution spatial data, they suffer from a fundamental flaw: line-of-sight. If a pedestrian walks behind a van or turns a corner, the camera loses them. This leads to incomplete trajectories and dangerous "blind spots."

On the other hand, mobile devices (phones, wearables) emit signals—like IMU (Inertial Measurement Unit) data, Wi-Fi, and GPS—that aren't blocked by physical obstacles. However, this "mobile modality" is notoriously noisy and lacks "layout" information; a phone can tell you it's moving, but it doesn't know the size of the person carrying it or their exact 3D bounding box in a camera's coordinate system.

The authors tackle the challenge of combining these two imperfect worlds: the precise-but-occludable visual modality and the persistent-but-noisy mobile modality.

Methods

The core of the paper is LTrajDiff, a framework built on Denoising Diffusion Probabilistic Models. Instead of trying to regress a trajectory directly, the model learns to "denoise" a sequence of layout points, guided by mobile and visual cues.

Key architectural innovations include:

  • Random Mask Strategy (RMS): During training, the model randomly masks out chunks of the visual data. This forces the network to learn how to rely on mobile signals to "fill in the blanks" when the camera is obstructed.
  • Siamese Masked Encoding Module: This contains two specialized sub-modules:
  • Temporal Alignment Module (TAM): Synchronizes the high-frequency noisy mobile signals with the visual frames.
  • Layout Extracting Module (LEM): This is the "spatial brain." It infers the object's physical dimensions (size/depth) from the few visual frames available and applies that knowledge to the parts of the trajectory where the object is invisible.
  • Modality Fusion Module (MFM): A cross-modal gate that merges features from both sensors into a unified representation for the diffusion decoder.

By using a coarse-to-fine diffusion strategy, the model iteratively refines its prediction, starting from random noise and settling into a precise 3D layout sequence.

Datasets

The researchers evaluated LTrajDiff on two primary multi-modal datasets:

  1. Vi-Fi Multi-modal Dataset: A benchmark specifically designed for vision-wireless fusion, containing synchronized video and smartphone sensor data (Wi-Fi, IMU).
  2. H3D Dataset: A large-scale full-surround 3D multi-object detection and tracking dataset used to test the model's performance in complex urban environments.

Key Insights and Ideas

  • Mobile as a Global Anchor: The mobile signal acts as a "rough map" that never goes dark. Even if it's noisy, it provides a continuous temporal thread that prevents the model from losing the object entirely during long occlusions.
  • Vision as a Spatial Template: You don't need to see a person for the whole trip to know how big they are. The model uses the "In-Sight" moments to learn the object's "layout" (bounding box size) and then projects that template onto the noisy mobile path during "Out-of-Sight" moments.
  • Diffusion for Imputation: Traditional Seq2Seq models often struggle with long-term occlusions, leading to "drift." Because diffusion models are generative, they are much better at maintaining structural consistency (like keeping the bounding box size stable) while hallucinating the missing parts of a path.

Key Results

LTrajDiff significantly outperformed state-of-the-art baselines like LSTMs, Transformers, and UNets, particularly in "extreme" scenarios:

  • Random Obstructions: On the Vi-Fi dataset, LTrajDiff achieved an IoU-D (Intersection over Union with Depth) of 0.69, compared to just 0.43 for the next best baseline (UNet).
  • Extremely Short Inputs: In a stress test where the model was only given 1/50th of the visual sequence (essentially a single frame), it could still reconstruct a plausible 3D trajectory. Baselines like LSTM and ViTag completely failed to converge in this setting.
  • Quantitative Accuracy: On the H3D dataset, the model reduced the Mean Square Error per Timestamp (MSE-T) to 2.72, nearly half the error of standard Transformer-based predictors (5.17).

Limitations and Future Work

The authors identify a few areas for improvement:

  • Inference Latency: Because diffusion models require multiple iterative steps ($K$ forward passes) to generate a result, they are currently slower than simple regression models. This is a hurdle for real-time autonomous driving.
  • 3D Representation: The current model predicts bounding boxes and depth. The authors suggest that moving toward full 3D mesh representations or real-time volume segmentations could provide even richer scene understanding.
  • Acceleration: Future work will likely focus on model pruning and sampling acceleration techniques to bring LTrajDiff closer to real-time deployment.

Summary updated

Abstract

Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. Nevertheless, real-world situations often involve obstructed cameras, missed objects, or objects out of sight due to environmental factors, leading to incomplete or noisy trajectories. To overcome these limitations, we propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount out-of-sight constraints, albeit introducing new challenges such as modality fusion, noisy data, and the absence of spatial layout and object size information. We employ a denoising diffusion model to predict precise layout sequences from noisy mobile data using a coarse-to-fine diffusion strategy, incorporating the Random Mask Strategy, Siamese Masked Encoding Module, and Modality Fusion Module. Our model predicts layout sequences by implicitly inferring object size and projection status from a single reference timestamp or significantly obstructed sequences. Achieving state-of-the-art results in randomly obstructed experiments, our model outperforms other baselines in extremely short input experiments, illustrating the effectiveness of leveraging noisy mobile data for layout sequence prediction. In summary, our approach offers a promising solution to the challenges faced by layout sequence and trajectory prediction models in real-world settings, paving the way for utilizing sensor data from mobile phones to accurately predict pedestrian bounding box trajectories. To the best of our knowledge, this is the first work that addresses severely obstructed and extremely short layout sequences by combining vision with noisy mobile modality, making it the pioneering work in the field of layout sequence trajectory prediction.
BibTeX
@inproceedings{zhang2023layout,
  author = {Haichao Zhang and Yi Xu and Hongsheng Lu and Takayuki Shimizu and Yun Fu},
  title = {{Layout Sequence Prediction From Noisy Mobile Modality}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611936},
}

Loading related content...

Figure