Masked Modeling for Human Motion Recovery under Occlusions

Zhiyin Qian, Siwei Zhang, Bharat Lal Bhatnagar, Federica Bogo, Siyu Tang

TL;DR

This paper introduces MoRo, a novel framework for reconstructing 3D human motion from monocular videos, even when parts of the body are occluded. MoRo uses a "masked modeling" approach, inspired by recent advances in language and image generation, to robustly handle missing information and achieve real-time performance.

What Problem Does This Paper Address?

Reconstructing accurate 3D human motion from single-camera videos is crucial for applications like augmented reality, robotics, and animation. However, real-world scenarios often involve frequent occlusions (e.g., a person partially hidden behind an object), which significantly degrade the performance of existing methods. While some approaches can handle occlusions, they are often slow, require extensive preprocessing, or are fragile to initial estimation errors. This paper aims to bridge this gap by developing an occlusion-robust, end-to-end generative model that can efficiently recover human motion from videos in real-time.

Methods

MoRo tackles the occlusion problem by reformulating motion reconstruction as a video-conditioned generative task using masked modeling. This technique, inspired by models like BERT, involves randomly masking parts of the input sequence (in this case, human motion data) and training the model to predict the missing parts. This inherently makes the model robust to missing information.

To overcome the scarcity of paired video-motion data, MoRo employs a cross-modality learning scheme:

  1. Trajectory-aware Motion Prior: A motion prior is trained on large motion capture (MoCap) datasets. This prior learns natural human dynamics and is designed to be trajectory-aware, meaning it understands how local body movements relate to overall global movement.
  2. Image-conditioned Pose Prior: An image-conditioned pose prior is trained on image-pose datasets. This helps the model learn diverse poses that might not be present in MoCap data and also provides a coarse global trajectory estimate from visual input.
  3. Video-conditioned Masked Transformer: A central transformer network fuses the information from the motion and pose priors with visual cues from the video. This network is fine-tuned on video-motion datasets to learn how to integrate these different modalities for robust motion recovery, especially under occlusions.

The model uses a multi-step inference process where it iteratively refines the predicted motion by re-masking and regenerating pose tokens based on confidence scores.

Datasets

MoRo is trained using a multi-stage approach on diverse datasets:

  • Motion Pretraining: AMASS (Archive of Motion Capture as Surface Shapes) dataset is used for training the trajectory-aware motion prior.
  • Image Pretraining: Standard image datasets like Human3.6M, MPI-INF-3DHP, COCO, and MPII are used to pretrain the image encoder and cross-modal decoder.
  • Video Fine-tuning: The full model is fine-tuned on two monocular video-motion datasets:
  • EgoBody: Captures human interactions in various 3D environments with a significant amount of occlusion.
  • RICH: Also captures human interactions in 3D environments, serving as a standard benchmark for video-based motion reconstruction.

The paper also introduces a curated subset of EgoBody, EgoBody-Occ, specifically for evaluating performance under severe occlusions.

Key Insights and Ideas

  • Masked Modeling for Motion: Applying masked modeling, successful in NLP and vision, to human motion reconstruction proves highly effective for handling occlusions. It allows the model to "fill in the blanks" naturally.
  • Cross-Modality Fusion is Key: The scarcity of paired video-motion data is a major bottleneck. MoRo's strategy of learning separate priors from MoCap and image-pose data, and then fusing them with video input, is a powerful way to leverage diverse data sources.
  • Trajectory-Aware Motion Prior: Explicitly modeling the relationship between local pose and global trajectory in the motion prior significantly improves temporal consistency and robustness.
  • Iterative Refinement for Occlusions: The multi-step inference process, where the model iteratively refines its predictions based on confidence, is crucial for gradually recovering motion in challenging occluded regions.
  • Unified Output: Unlike some prior methods that produce separate camera-space and world-grounded outputs, MoRo aims for a single, unified trajectory that balances visual alignment and motion realism.

Key Results

  • Superior Performance on Occlusions: MoRo significantly outperforms state-of-the-art methods on the EgoBody-Occ dataset, achieving a 16% and 31% MPJPE improvement over the best baseline for visible and occluded joints, respectively. It also shows a 58% improvement in global joint reconstruction (GMPJPE).
  • Real-time Performance: MoRo achieves real-time inference at 70 FPS on a single H200 GPU, a significant advantage over slower optimization or diffusion-based methods.
  • Competitive Performance in Non-Occluded Scenarios: On the RICH dataset, MoRo performs comparably to baselines while yielding more plausible motion (lower acceleration and jitter).
  • Robustness to Truncated Inputs: The confidence-guided masking strategy during training improves robustness to truncated bounding boxes.

Limitations and Future Work

  • Static Camera Assumption: The current method is restricted to static camera setups with known intrinsics. This limits its applicability to videos captured by moving cameras.
  • Future Work: The authors plan to incorporate techniques for modeling camera motion to extend MoRo to dynamic camera scenarios.

Summary updated

Abstract

Human motion reconstruction from monocular videos is a fundamental challenge in computer vision, with broad applications in AR/VR, robotics, and digital content creation, but remains challenging under frequent occlusions in real-world settings.Existing regression-based methods are efficient but fragile to missing observations, while optimization- and diffusion-based approaches improve robustness at the cost of slow inference speed and heavy preprocessing steps. To address these limitations, we leverage recent advances in generative masked modelling and present MoRo: Masked Modeling for human motion Recovery under Occlusions. MoRo is an occlusion-robust, end-to-end generative framework that formulates motion reconstruction as a video-conditioned task, and efficiently recover human motion in a consistent global coordinate system from RGB videos. By masked modelling, MoRo naturally handles occlusions while enabling efficient, end-to-end inference. To overcome the scarcity of paired video–motion data, we design a cross-modality learning scheme that learns multi-modal priors from a set of heterogeneous datasets: (i) a trajectory-aware motion prior trained on MoCap datasets, (ii) an image-conditioned pose prior trained on image-pose datasets, capturing diverse per-frame poses, and (iii) a video-conditioned masked transformer that fuses motion and pose priors, finetuned on video–motion datasets to integrate visual cues with motion dynamics for robust inference. Extensive experiments on EgoBody and RICH demonstrate that MoRo substantially outperforms state-of-the-art methods in accuracy and motion realism under occlusions, while performing on-par in non-occluded scenarios. MoRo achieves real-time inference at 70 FPS on a single H200 GPU. Project page: https://mikeqzy.github.io/MoRo.
BibTeX
@inproceedings{qian2026masked,
  author = {Zhiyin Qian and Siwei Zhang and Bharat Lal Bhatnagar and Federica Bogo and Siyu Tang},
  title = {{Masked Modeling for Human Motion Recovery under Occlusions}},
  booktitle = {3DV},
  year = {2026},
  doi = {10.1109/3dv69130.2026.00074},
}
Figure