Abstract
Although facial landmark detection (FLD) has gained significant progress, existing FLD methods still suffer from performance drops on partially non-visible faces, such as faces with occlusions or under extreme lighting conditions or poses. To address this issue, we introduce ORFormer, a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically, ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way, the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings, enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features, OR-Former compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods, our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.
Introduction
Facial Landmark Detection (FLD)—the task of localizing key points like eyes, nose, and mouth—is a cornerstone of face recognition and augmented reality. While deep learning has made FLD highly accurate for clear, front-facing photos, performance often plummets when faces are partially hidden by hands, sunglasses, or deep shadows.
Most existing methods either ignore occluded regions, leading to corrupted features, or require expensive manual labels that explicitly mark which parts of a face are hidden. ORFormer addresses this by providing a unified framework to both detect and "repair" these non-visible regions automatically.
Methodology: The Messenger Token Mechanism
The core innovation of ORFormer is the Messenger Token mechanism, which allows the model to identify and recover corrupted local information using global context.
1. Quantized Heatmap Prior
Before training the main model, a quantized autoencoder (similar to a VQVAE) is trained on clean, unoccluded faces. This creates a "codebook" of healthy facial features, providing a structural prior that the model can use to reconstruct missing parts.
2. Messenger Tokens ($M_i$)
For every image patch token ($X_i$), the model assigns a corresponding messenger token. While the patch token focuses on local data, the messenger token uses cross-attention to aggregate information from every other patch except its own. This acts as a "consensus" mechanism: if the rest of the face suggests an eye should be present, but the local patch is obscured, the disparity between the patch token and its messenger token signals an occlusion.
3. Occlusion Detection and Recovery
By comparing the regular patch embedding with its messenger counterpart, the model generates an occlusion map ($\alpha$). If a patch is flagged as occluded, the model replaces or blends its corrupted features with the consensus features provided by the messenger token. This effectively "hallucinates" the missing parts based on the face's overall geometry.
Evaluation and Results
ORFormer was evaluated on three major benchmarks: WFLW (98 landmarks, including occlusion/pose subsets), COFW (the gold standard for occlusion robustness), and 300W.
Quantitative Performance
ORFormer achieved state-of-the-art (SOTA) performance across the board:
- WFLW: Achieved a 3.86 NME (Normalized Mean Error), outperforming previous leaders like LDEQ and STAR.
- COFW: Achieved a 4.46 NME, demonstrating superior robustness on faces with heavy physical obstructions.
Qualitative Success
Visualizations demonstrate the model's ability to accurately place landmarks even when features are completely covered by hands or objects—scenarios where baseline models typically fail.
Conclusion
ORFormer introduces a robust way to handle non-visible facial regions by leveraging global context through messenger tokens. While it excels at recovering occluded features, future work may focus on improving performance for extreme facial expressions, where features are deformed rather than hidden. The current architecture provides a powerful enhancement for existing FLD pipelines, significantly improving accuracy in real-world, "in-the-wild" scenarios.
Summary updated