Temporal Amodal Pose Completion

32 concepts26 papers1 proposal

When tracking human or animal movement, occlusion is an unavoidable reality. Whether an arm disappears behind a torso (self-occlusion) or a subject passes behind an environmental obstacle, standard frame-by-frame pose estimation models face a fundamental limitation: they cannot estimate what they cannot see. When pixels are missing, single-frame models are forced to make unconstrained guesses.

Temporal amodal pose completion solves this by shifting the problem from spatial estimation to temporal inference. Instead of treating a video as a series of isolated snapshots, this paradigm treats human motion as a continuous, physically constrained trajectory. When a joint is hidden in the current frame, its position is not truly random; it is tightly bounded by its historical momentum, its future destination, and the structural constraints of the skeleton. By leveraging the temporal dimension, modern architectures can "see through" occlusions, reconstructing complete 3D poses from incomplete visual data.

Routing Information Across Time via Attention

The most direct way to resolve an occlusion is to find a frame where the hidden body part is visible and borrow its features. Temporal self-attention mechanisms excel at this by treating a video sequence as a fully connected graph where information can be dynamically routed across time.

[Frame t-5: Visible]  ---> (Attention Weight) ---> [Frame t: Occluded]
[Frame t+5: Visible]  ---> (Attention Weight) ---> [Frame t: Occluded]

In a purely transformer-based architecture, such as the one proposed by Ce Zheng et al. in 3D Human Pose Estimation with Spatial and Temporal Transformers, the network decouples spatial joint relationships from temporal frame correlations. When a joint's visual features disappear, the temporal transformer layer computes high attention weights for surrounding frames where the joint is visible, effectively interpolating the missing trajectory based on global context.

This mechanism allows models to treat occlusions as a sequence-repair task. For example, Qiongjie Cui et al. developed a framework in Towards Accurate 3D Human Motion Prediction From Incomplete Observations that applies temporal self-attention directly to noisy, incomplete sequences, repairing missing joint data before predicting future motion. Similarly, Yilin Wen et al. demonstrate in Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos that short-term temporal attention can dynamically shift focus to adjacent frames to resolve severe self-occlusions during egocentric human-object interactions.

To make this process explicit, some architectures introduce specialized tokens or indicators to flag missing data. Stefan Becker et al. introduce "missing tokens" in MissFormer: (In-)Attention-Based Handling of Missing Observations for Trajectory Filtering and Prediction, which explicitly instruct the self-attention mechanism to ignore empty inputs and route information from valid observations instead. In a similar vein, Mehwish Ghafoor demonstrates in Quantification of Occlusion Handling Capability of a 3D Human Pose Estimation Framework that providing a network with a simple binary indicator of which joints are occluded allows extended temporal sequences to reliably reconstruct missing spatial data.

This temporal aggregation strategy is so powerful that its underlying intuition generalizes to entirely different domains. For instance, Corinne Stucker et al. demonstrate in U-TILISE: A Sequence-to-Sequence Model for Cloud Removal in Optical Satellite Time Series that temporal self-attention can seamlessly reconstruct missing geographic data hidden behind clouds by selectively weighting clear observations from completely different points in time.

State Persistence and Localized Recurrence

While attention mechanisms look globally across a time horizon, recurrent architectures maintain a continuous internal state, tracking the momentum and trajectory of the subject frame by frame. This is particularly valuable for handling long-term or progressive occlusions where a body part remains hidden for an extended period.

A naive recurrent model attempts to track the entire body state at once, which can cause independent limb movements to interfere with one another. To overcome this, Srikrishna Karanam et al. propose a localized approach in Learning Local Recurrent Models for Human Mesh Recovery. By dividing the human body into distinct parts and assigning a dedicated recurrent model to each limb, the system preserves the independent temporal history of each part, allowing it to correctly infer a limb's position even when the subject turns completely away from the camera.

                  ---> [Local RNN: Left Arm]  ---> State Persistence
[Full Body Mesh]  ---> [Local RNN: Right Leg] ---> State Persistence
                  ---> [Local RNN: Torso]     ---> State Persistence

Alternatively, models can store past observations in an explicit memory buffer. In Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering, YoungJoong Kwon et al. utilize a "skeletal feature bank" that stores representations from previous time steps. When a body part becomes occluded, the system queries this feature bank to retrieve past visual states, resolving current ambiguities.

Whether using explicit feature banks or gated architectures like Gated Recurrent Units (GRUs)—as explored by Chengcheng Tang et al. in UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality—the core mechanism remains the same: leveraging the physical momentum of previous frames to synthesize a plausible trajectory through periods of darkness.

Enforcing Kinematic and Interaction Priors

Temporal context alone can fail if the occlusion lasts too long or if the visual data is highly degraded. To prevent the model from generating physically impossible or disjointed poses, temporal continuity must be paired with structural regularizers: kinematic and interaction priors.

Kinematic priors enforce the rules of anatomy. In AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild, Daniel Joska et al. introduce a trajectory optimization method that outperforms standard frame-based triangulation by enforcing rigid skeletal constraints alongside temporal continuity across the entire motion sequence. When tracking subjects in the wild, these constraints ensure that occluded limbs do not stretch or disconnect. Similarly, Hojun Jang et al. demonstrate in Dynamic Mesh Recovery from Partial Point Cloud Sequence that mapping noisy, occluded point clouds into a clean latent space trained on kinematic priors ensures smooth, realistic reconstructions.

Interaction priors extend this logic to the environment. When a human interacts with an object, the physics of the interaction dictates the motion of both entities. Jan Eric Lenssen et al. exploit this insight in InterTrack: Tracking Human Object Interaction without Object Templates. By analyzing a temporal window of human poses and image features, their system can infer the 3D trajectory of an object even when it is completely obscured by the user's body, proving that the visible motion of the actor serves as a powerful proxy for the hidden object.

Ultimately, by combining temporal attention, state persistence, and physical priors, amodal pose completion frameworks transform occlusion from a catastrophic failure point into a manageable inference problem. As shown by Shubham Goel et al. in Humans in 4D: Reconstructing and Tracking Humans with Transformers, training models with explicit temporal masking objectives allows them to reliably predict full 3D human meshes through total occlusions, leading to significantly more robust tracking in real-world environments.

Key Foundational Work

UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality (2022) — contributed 2 concept elements to this direction.
Towards Accurate 3D Human Motion Prediction From Incomplete Observations (2021) — contributed 2 concept elements to this direction.
Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos (2022) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_occlusion_robust_pose_estimation_amodal__9b57473f,
  title = {occlusion-robust pose estimation (amodal pose completion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/occlusion-robust-pose-estimation-amodal-pose-completion/rdr_9b57473fb59342378cb866f6017b5b78},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_occlusion_robust_pose_estimation_amodal__9b57473f_v1,
  title = {occlusion-robust pose estimation (amodal pose completion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/occlusion-robust-pose-estimation-amodal-pose-completion/rdr_9b57473fb59342378cb866f6017b5b78/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake