Temporal Continuity for Occluded 3D Human Pose Estimation

This direction is split into other directions: occlusion-robust pose estimation (amodal pose completion). Its last version stays readable and citable here.

Version 1 of 193 papers107 conceptsfrozen 2026-10-04

In the world of computer vision, reconstructing a 3D human figure from a 2D video is a bit like solving a puzzle where half the pieces are constantly being stolen. As people move, their limbs disappear behind their own bodies (self-occlusion), hide behind furniture, or vanish into the blur of rapid motion.

Traditional "frame-by-frame" estimators treat each video frame as a standalone photograph. When a joint disappears, these models are forced to guess its location based only on the visible pixels. This leads to the "jitter" problem: a knee might inexplicably pop into existence three feet from where it was a millisecond ago, or a person's body shape might change from frame to frame.

The research direction emerging from recent work suggests a more intuitive solution: temporal continuity. Instead of treating a video as a sequence of static images, researchers are treating human movement as a continuous, physically constrained path through space. By leveraging the "memory" of where a limb was and the "expectation" of where it is going, these systems can "see" joints that are technically invisible.

The Grammar of Movement

The fundamental insight shared by researchers like Davis Rempe and Ho‐Jun Jang is that human motion follows a specific "grammar." Our joints don't move randomly; they follow the laws of physics and biology.

In HuMoR: 3D Human Motion Model for Robust Pose Estimation, Davis Rempe and colleagues introduced a generative model that learns these motion transitions. Instead of predicting a static pose, the model predicts the change from one pose to the next. This acts as a powerful "motion prior." If a person is walking and their left leg is obscured by a desk, the model doesn't need to see the leg to know where it is; the "grammar" of walking dictates exactly where that leg must be to maintain balance and momentum.

Similarly, Ho‐Jun Jang developed ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening. By pre-training a transformer on massive amounts of clean 3D motion capture data, the system learns to "fill in the blanks" (inbetweening) when sensor data—whether from cameras, LiDAR, or IMUs—becomes noisy or intermittent.

Transformers as Temporal Bridges

While early attempts at temporal consistency used Recurrent Neural Networks (RNNs) to maintain a memory of previous frames, the field has largely shifted toward Transformer architectures. Transformers allow a model to look at a "window" of time—both past and future—to resolve current ambiguities.

Ce Zheng and Sijie Zhu pioneered this with PoseFormer, the first purely transformer-based approach for 3D pose. By decoupling spatial relationships (how the hand relates to the elbow) from temporal correlations (how the elbow moves over 81 frames), PoseFormer can infer hidden joint positions by analyzing the trajectory of the movement.

This concept was pushed further by Fabien Baradel and Grégory Rogez in PoseBERT. They applied the "masked modeling" logic of BERT to human motion. Just as BERT learns to predict a missing word in a sentence by looking at the surrounding context, PoseBERT learns to "hallucinate" missing joints by looking at the frames before and after the occlusion. This allows the system to maintain a physically plausible skeleton even during periods of total visual dropout.

Seeing the Invisible in Egocentric Video

One of the most extreme tests for temporal continuity is egocentric (first-person) video, where the wearer's own body is often completely outside the camera's field of view.

In Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video, Kristen Grauman and Hao Jiang 0007 demonstrated that you can reconstruct a full body pose even when the limbs are "invisible." They achieved this by treating the person's movement as a global optimization problem—a "pose path." By analyzing how the background moves (indicating the wearer's head motion) and matching that to a library of known human transitions, the system can infer the most likely leg and arm positions.

More recently, Jiaxi Jiang and Christian Holz addressed this for Mixed Reality headsets in EgoPoser. Because headsets only "see" hands intermittently, EgoPoser uses temporal history to maintain a stable avatar. When a hand leaves the camera's view, the model doesn't just drop the tracking; it uses the previous trajectory to keep the avatar's arm in a realistic position until the hand returns.

Mutual Learning and Global Dynamics

A final facet of this direction is the integration of global motion cues, such as optical flow, to supplement local joint detection.

In Joint-Motion Mutual Learning for Pose Estimation in Videos, Haipeng Chen and Yifang Yin proposed JM-Pose. This framework forces local joint data (heatmaps) and global motion dynamics (optical flow) to "talk" to each other. When a joint is blurred or occluded, the global motion trajectory acts as a predictive guide, allowing the model to reconstruct the missing spatial information.

This shift from "detecting" to "reasoning" marks a transition in the field. By treating time as a fundamental dimension of the human body, researchers are moving toward systems that don't just see what is there, but understand the underlying rhythm of how we move.

Papers in this version

Showing 60 of 93 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_continuity_for_occluded_3d_huma_df585a92,
  title = {Temporal Continuity for Occluded 3D Human Pose Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-continuity-for-occluded-3d-human-pose-estimation/rdr_df585a92f95b41c5ba67f7dfd4722740},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_continuity_for_occluded_3d_huma_df585a92_v1,
  title = {Temporal Continuity for Occluded 3D Human Pose Estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-continuity-for-occluded-3d-human-pose-estimation/rdr_df585a92f95b41c5ba67f7dfd4722740/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}