Landmark estimation—the task of localizing specific semantic keypoints on an object—serves as the foundation for applications ranging from facial expression synthesis to surgical navigation. Whether tracking the corners of a moving mouth or aligning a pre-operative scan to a patient's spine, algorithms must map visual data to precise geometric coordinates.
However, standard landmark estimation frameworks suffer from a fundamental vulnerability: occlusion sensitivity. In real-world environments, the line of sight to key landmarks is frequently interrupted by hands, accessories, surgical tools, or extreme pose changes. When these features disappear, local appearance-based models break down, leading to distorted coordinate predictions and tracking failure.
To build systems that can withstand heavy occlusion, researchers have shifted away from treating landmarks as independent pixel patches. Instead, modern approaches leverage global geometric priors, structural topologies, and 3D deformations to infer the positions of invisible features from the visible ones.
The Mechanism of Failure: Why Occlusion Breaks Tracking
To understand why occlusion is so disruptive, consider how a typical regression-based landmarker works. The model learns a direct mapping from local image patches to coordinate spaces. When an object blocks a landmark, the local appearance changes completely. A mouth covered by a hand no longer contains the edge gradients of lips; instead, it contains the texture of skin or fingers.
As Jie Shen demonstrated in The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results, physical occlusions and extreme head poses cause a much sharper decline in tracking accuracy than variations in lighting or facial expressions. When keypoints temporarily disappear, standard models struggle to maintain structural consistency or recover the target's position once it reappears.
This brittleness ripples across various domains:
- Facial Analysis and Editing: In geometry-driven face manipulation, coordinate predictors cannot reliably locate points hidden by hair or hands, which distorts downstream synthesis as noted by Zhenghao Zhang et al. in LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editing.
- In-Car Monitoring: Driver-monitoring systems that depend on explicit facial landmarks fail when drivers wear sunglasses or masks, disrupting continuous head pose estimation (Marco Venturelli et al. in Deep Head Pose Estimation from Depth Data for In-car Automotive Applications).
- Medical Augmented Reality: In markerless surgical tracking, if a patient's face or head is partially obscured by surgical drapes or tools, the coordinate alignment fails entirely, restricting the system's operational view (Hon Lung Ho et al. in Web-based Augmented Reality with Auto-Scaling and Real-Time Head Tracking towards Markerless Neurointerventional Preoperative Planning and Training of Head-mounted Robotic Needle Insertion).
Global Shape Priors as Geometric Anchors
The primary intuition for overcoming occlusion is simple: if you know the global structure of an object, you do not need to see every part of it to know where the hidden pieces are. If a model observes the eyes, nose, and jawline, it should be able to deduce the approximate location of an occluded mouth.
To operationalize this intuition, researchers embed global geometric constraints directly into neural network architectures. For example, Xiang Yu 0002 et al. introduced the Deep Deformation Network in Deep Deformation Network for Object Landmark Localization. Their architecture uses a Shape Basis Network to initialize a global geometric configuration, ensuring that even if large sections of an object are obscured, the model's coordinate predictions remain anchored to a plausible, non-distorted global geometry.
[Input Image with Occlusion]
│
▼
[Local Feature Extraction] ──► (Missing/Corrupted Local Patches)
│ │
▼ ▼
[Global Shape Prior / RBM] ──► [Inferred Hidden Geometry] ──► [Valid Landmark Output]
Similarly, modeling complex, non-linear shape transitions helps maintain tracking stability. Yue Wu 0002 et al. utilized Restricted Boltzmann Machines (RBMs) in Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines to capture global facial structures across varying poses. By relying on a robust geometric prior, their system estimates the positions of hidden landmarks based entirely on the visible portions of the face, preventing the tracking loop from diverging when features are blocked by hands or shadows.
Structural Topology and Graph Constraints
Instead of treating landmarks as a flat vector of coordinates, another powerful strategy is to model the structural relationships between landmarks as a graph or topology.
In FaceTopoNet: Facial Expression Recognition using Face Topology Learning, Alireza Sepas-Moghaddam et al. designed an end-to-end framework that automatically learns an optimal "tree topology" connecting different facial regions. By organizing landmarks into a structured hierarchy, the network gains structural flexibility. If a specific region (like the mouth) is occluded, the integrity of the remaining tree structure prevents the model from collapsing, allowing it to maintain downstream performance.
When landmarks are heavily obscured, the structural data fed into recognition networks inevitably becomes noisy or incomplete. To prevent this noise from corrupting downstream tasks, Zhiyu Wu et al. proposed LA-Net in LA-Net: Landmark-Aware Learning for Reliable Facial Expression Recognition under Label Noise. Their framework uses landmarks as structural anchors but introduces noise-tolerant contrastive losses to mitigate performance degradation when extreme head poses or heavy occlusions obscure those anchors.
Lifting to 3D and Occlusion Completion
While 2D tracking is highly sensitive to perspective changes and obstructions, lifting the tracking problem into 3D space provides an explicit physical model that inherently understands occlusion. A 3D object cannot occupy two spaces at once, and its rigid structure dictates exactly how features disappear behind occluding objects or during self-occlusion (rotation).
Pei-Lun Hsieh et al. tackled this in Unconstrained realtime facial performance capture by unifying face segmentation with an explicit occlusion completion loop. By identifying which pixels belong to occluding objects (like hair or hands), the system isolates the corrupted data and uses a 3D model to "fill in" the missing geometric information on the fly.
2D Image ──► [Occlusion Segmentation] ──► [Isolate Corrupted Pixels]
│
▼
3D Prior ───────────────────────────────► [3D Mesh Completion Loop] ──► Stable Tracking
Other frameworks combine 2D detection with iterative 3D reconstruction loops to enforce consistency. Shuang Liu 0001 et al. demonstrated in Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video that an iterative coarse-to-dense 3D reconstruction loop allows a system to maintain tracking stability even when visual landmarks are completely hidden. By anchoring the tracking to a 3D Morphable Model (3DMM), as also seen in the full-head reenactment work by Anastasios Roussos et al. in ReenactNet: Real-time Full Head Reenactment, the system ensures that transient obstructions do not cause catastrophic tracking loss or mesh distortion.
Beyond Faces: Semantic Contours in Medical Imaging
The challenge of occlusion sensitivity extends far beyond facial tracking. In clinical environments, automated measurement and registration techniques must locate anatomical landmarks to guide surgical robots or quantify trauma. However, these reference points are frequently obscured by overlapping tissues, hemorrhages, or surgical tools.
For instance, in Diffusion Model based Semi-supervised Learning on Brain Hemorrhage Images for Efficient Midline Shift Quantification, Shizhan Gong et al. note that severe brain trauma or hemorrhages can completely distort or obscure standard anatomical landmarks. To counter this, they use a diffusion model to learn a global structural prior of a "normal" brain, allowing the system to accurately quantify midline shifts even when traditional geometric landmarks are unrecognizable.
Point-Based Tracking: [X] (Tool Occlusion) ──► Tracking Fails
Contour-Based Tracking: ───[Tool Occlusion]─── ──► Contextual Alignment Succeeds
When discrete point landmarks are prone to total obstruction, shifting the target representation from sparse points to continuous semantic contours can provide a more resilient tracking signal. Ao Shen et al. explored this paradigm in Semantic edge-guided 2D/3D registration for vertebrae in radiographs. In spinal surgery radiographs, specific anatomical points are routinely blocked by surgical instruments. By training models to prioritize continuous semantic edges—such as the pedicles and vertebral sides—the system can tolerate partial occlusions. If a surgical tool blocks a portion of the bone, the remaining visible contour provides enough contextual geometric data to maintain sub-millimeter registration accuracy.
Key Foundational Work
Robust Real-Time Face Tracking and Gesture Recognition (1997) — contributed 1 concept elements to this direction.
Automatic non-rigid registration of 3D dynamic data for facial expression synthesis and transfer (2008) — contributed 1 concept elements to this direction.
The Use of Dynamic Deformable Templates for Lip Tracking in an Audio-Visual Corpus with Large Variations in Head Pose, Face Illumination and Lip Shapes (2008) — contributed 1 concept elements to this direction.