Latent World Modeling for Point-Cloud Video Pretraining

Zhuoyue Zhang, Jian Liu, Jihua Zhu, Ajmal Saeed Mian

Abstract

Masked modeling is the dominant self-supervised paradigm for point-cloud videos. However, it leaks the information it is meant to predict, whether it reconstructs masked points as in MAE-style methods or predicts their latent features. Because point-cloud tokens are anchored at points sampled from each frame, querying a target at its own position reveals where its geometry has moved, in reconstruction and latent prediction alike. For latent predictors, random spatio-temporal masks introduce a further shortcut, as visible frames on both sides of each target allow interpolation rather than forecasting. The pretext task thus reduces to completion. Once the target positions are withheld, a masked latent predictor performs only slightly better than copying the last frame. To remove this leakage, we propose PointWM, a latent world model that turns completion into forecasting by predicting the features and geometry of future frames from strictly earlier context. Each target is queried only with its last-seen position and timestamp, while a latent action learned without annotations is meant to capture the part of the future that the past leaves undetermined. On HOI4D action segmentation with a P4Transformer backbone, PointWM improves F1@50 over training from scratch by 24.9 offline and 29.1 under strictly causal inference. These results show the promise of latent world models as a leakage-free pretraining paradigm for point-cloud videos.Our code is publicly available at https://github.com/InitalZ/PointWM.
BibTeX
@misc{zhang2026latent,
  author = {Zhuoyue Zhang and Jian Liu and Jihua Zhu and Ajmal Saeed Mian},
  title = {{Latent World Modeling for Point-Cloud Video Pretraining}},
  year = {2026},
  eprint = {2611.20090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20090},
}
Figure