Latent World Modeling for Point-Cloud Video Pretraining

Zhuoyue Zhang, Jian Liu, Jihua Zhu, Ajmal Saeed Mian

TL;DR

PointWM is a novel self-supervised pretraining framework that fixes a critical flaw in point-cloud video models: "positional leakage." By forcing models to forecast future geometry and features from past context—rather than just filling in the blanks—it achieves massive performance gains in action segmentation, particularly for real-time, causal applications.

What Problem Does This Paper Address?

In the world of self-supervised learning, Masked Autoencoders (MAE) are king. The idea is simple: hide parts of the data and ask the model to reconstruct them. While this works wonders for images and standard video, it hits a snag with point clouds.

Unlike RGB video, where pixels sit on a fixed grid, point-cloud tokens are anchored to specific 3D coordinates sampled from the scene's geometry. The authors point out a "shortcut" that current models exploit: if you ask a model to predict a feature at a specific future coordinate, you are essentially telling the model exactly where the objects have moved. This "positional leakage" turns a difficult forecasting task into a trivial completion task. Furthermore, random temporal masking allows models to "cheat" by interpolating between visible frames rather than truly predicting the future.

Methods

The authors introduce PointWM (Point-cloud latent World Model), which shifts the paradigm from completion to leakage-free forecasting.

Key components include:

  • Causal Masking: The model only sees frames from the past to predict the future, preventing temporal interpolation.
  • Last-Seen Queries: Instead of using ground-truth future positions as queries, PointWM uses the coordinates from the last visible frame. The model must then predict both the latent features and the geometric displacement (how the points moved).
  • Latent Action Modeling: Because the future is inherently uncertain (a hand could move left or right), the model learns an annotation-free "latent action." An Inverse Dynamics Model (IDM) infers the action from the full sequence during training, while a Prior learns to guess that action based only on the past context.
  • Multi-Branch Architecture:
    • Target Branch: Uses an EMA (Exponential Moving Average) teacher to provide stable latent targets.
  • Context Branch: The student model, seeing only the past.
  • Observation Branch: The student model seeing the full clip to help train the latent action posterior.
  • Regularization: To prevent the model from simply outputting constant values (representation collapse), they employ VICReg (Variance-Invariance-Covariance Regularization).

Datasets

The researchers evaluated PointWM on two primary benchmarks:

  1. HOI4D: A large-scale egocentric dataset focusing on hand-object interactions. It contains nearly 3,000 videos with 19 action categories.
  2. MSR-Action3D: A dataset of 20 full-body actions (e.g., jogging, waving). This was used primarily to test how well features learned on one type of motion (hands) transfer to another (full bodies).

Key Insights and Ideas

  • Positions are Information: The most profound insight is that in 3D point clouds, geometry is the signal. Withholding future positions forces the model to build a mental map of dynamics—a "world model"—rather than just a sophisticated lookup table.
  • Forecasting > Completion: The paper demonstrates that a model trained to forecast (predicting $T+1$ from $T$) develops much more robust features for downstream tasks than one trained to complete (filling a hole at $T$ using $T-1$ and $T+1$).
  • Causal Strength: PointWM shows its greatest strength in "strictly causal" settings. This is vital for robotics and AR/VR, where a system must understand an action as it happens, without the luxury of seeing the "future" frames of a pre-recorded clip.

Key Results

PointWM sets a new bar for self-supervised pretraining in this domain:

  • Action Segmentation (HOI4D): Using a P4Transformer backbone, PointWM improved the F1@50 score by 24.9 points (offline) and a staggering 29.1 points (strictly causal) compared to training from scratch.
  • Superiority over Leaky Models: It outperformed "leaky" masked latent predictors by 15.0 F1 points when only 10% of labels were available, proving that the features learned are significantly more meaningful.
  • Cross-Dataset Transfer: Pretraining on hand-object interactions (HOI4D) improved accuracy on full-body action recognition (MSR-Action3D) by over 4%, suggesting the model learns general principles of 3D motion.

Limitations and Future Work

  • The Prior-Posterior Gap: While the latent action prior helps, there is still a gap between what the model can guess from the past and what actually happens (the "oracle" posterior). Closing this gap is a key area for future research.
  • Approximate Correspondence: Because points are resampled in every frame, the model doesn't have perfect "point identities" over time. It tracks surface regions approximately, which might struggle with extremely fast or complex deformations.
  • Data Scaling: The current work focuses on HOI4D. The authors suggest that scaling this "world modeling" approach to even larger and more diverse datasets is the logical next step.

Summary updated

Abstract

Masked modeling is the dominant self-supervised paradigm for point-cloud videos. However, it leaks the information it is meant to predict, whether it reconstructs masked points as in MAE-style methods or predicts their latent features. Because point-cloud tokens are anchored at points sampled from each frame, querying a target at its own position reveals where its geometry has moved, in reconstruction and latent prediction alike. For latent predictors, random spatio-temporal masks introduce a further shortcut, as visible frames on both sides of each target allow interpolation rather than forecasting. The pretext task thus reduces to completion. Once the target positions are withheld, a masked latent predictor performs only slightly better than copying the last frame. To remove this leakage, we propose PointWM, a latent world model that turns completion into forecasting by predicting the features and geometry of future frames from strictly earlier context. Each target is queried only with its last-seen position and timestamp, while a latent action learned without annotations is meant to capture the part of the future that the past leaves undetermined. On HOI4D action segmentation with a P4Transformer backbone, PointWM improves F1@50 over training from scratch by 24.9 offline and 29.1 under strictly causal inference. These results show the promise of latent world models as a leakage-free pretraining paradigm for point-cloud videos.Our code is publicly available at https://github.com/InitalZ/PointWM.
BibTeX
@misc{zhang2026latent,
  author = {Zhuoyue Zhang and Jian Liu and Jihua Zhu and Ajmal Saeed Mian},
  title = {{Latent World Modeling for Point-Cloud Video Pretraining}},
  year = {2026},
  eprint = {2611.20090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20090},
}
Figure