ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection

Jui-Che Chiang, Hou-Ning Hu, Bo-Syuan Hou, Chia-Yu Tseng, Yu-Lun Liu, Min-Hung Chen, Yen-Yu Lin

Abstract

Although facial landmark detection (FLD) has gained significant progress, existing FLD methods still suffer from performance drops on partially non-visible faces, such as faces with occlusions or under extreme lighting conditions or poses. To address this issue, we introduce ORFormer, a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically, ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way, the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings, enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features, ORFormer compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods, our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.

Introduction

Facial Landmark Detection (FLD)—the task of pinpointing coordinates for features like eyes, nose, and mouth—is a foundational step for numerous computer vision applications, ranging from face filters and AR to driver fatigue monitoring. However, standard models frequently struggle when parts of the face are non-visible, such as when a hand covers a mouth, a mask is worn, or deep shadows obscure features.

Existing solutions often rely on explicit occlusion labels, which are expensive and time-consuming to annotate, or require multiple video frames to "see" behind an object. Furthermore, standard Transformers often propagate "noise" from corrupted regions through their self-attention mechanism, leading to inaccurate landmark predictions. ORFormer addresses these challenges by autonomously detecting and recovering missing facial parts from a single image without requiring additional labels.

Methodology

The authors propose a robust two-stage approach to handle occlusions by leveraging facial priors and a novel transformer architecture.

1. Quantized Heatmap Generator

The model first learns a representation of "clean" facial structures. Using a VQVAE-style approach, it trains an encoder-decoder to represent facial edge heatmaps using a discrete Codebook. This codebook acts as a structural prior, storing high-quality, unoccluded facial feature patterns that the model can later reference to reconstruct missing data.

2. The ORFormer Architecture

The core innovation lies in the Occlusion-Robust Transformer (ORFormer). For every standard image patch token ($X_i$), the architecture introduces a corresponding Messenger Token ($M_i$).

  • Messenger Tokens: These tokens use cross-attention to aggregate information from all patches except their own. Essentially, $M_i$ represents the "consensus" of the rest of the face regarding what should be present in patch $i$.
  • Occlusion Detection: By comparing the original patch $X_i$ to its messenger $M_i$, the model calculates a disparity. A high disparity suggests that the patch is likely occluded or corrupted.
  • Feature Recovery: The model generates an occlusion map ($\alpha$). It then reconstructs the final feature by blending the original patch and the messenger's consensus based on $\alpha$. If a patch is detected as occluded, the messenger's information—derived from the visible parts of the face—takes precedence.
The two-stage framework: (a) training the quantized heatmap generator to learn a codebook prior, and (b) the inference pipeline where ORFormer recovers features to generate robust heatmaps.

3. Integration with FLD Backbones

The "recovered" high-fidelity heatmaps produced by ORFormer are integrated into existing landmark detection backbones, such as Hourglass networks. This provides a cleaner, more robust signal for the final coordinate prediction stage.

Key Insights

  • The "Consensus" Strategy: The model operates on the principle that the visible parts of a face contain enough contextual information to infer the location of missing parts. For example, if the left eye and chin are visible, the messenger token can infer the position of an occluded right eye.
  • Occlusion-Aware Attention: To prevent the propagation of noise, messenger tokens are designed to ignore other patches that are also detected as occluded. This ensures the recovery process is powered exclusively by reliable, visible data.
  • Codebook as a Prior: By using a quantized codebook, the model does not merely guess pixels; it selects from a library of mathematically valid facial structures learned during pre-training on clean data.

Experimental Results

ORFormer was evaluated on three challenging benchmarks: WFLW (diverse subsets like "Expression" and "Illumination"), COFW (specifically designed for occlusion robustness), and 300W.

  • State-of-the-Art Performance: On WFLW, ORFormer achieved a Normalized Mean Error (NME) of 3.86, outperforming previous SOTA models like LDEQ and STAR. On the heavily occluded COFW dataset, it achieved an NME of 4.46.
  • Qualitative Superiority: Visualizations demonstrate that while baseline models like STAR "drift" when a face is covered by a mask or hand, ORFormer’s landmarks remain locked to the correct anatomical positions.
  • Computational Efficiency: Despite the additional tokens, the model is efficient. When integrated into a 4-stack Hourglass network, it reduced Mult-Add operations by 20.6% and inference time by 15.9% compared to a 5-stack baseline, while delivering superior accuracy.
Qualitative comparison of landmark detection under heavy occlusion. ORFormer (Ours) maintains anatomical accuracy where previous methods like STAR fail, as highlighted in the yellow ellipses.

Conclusion and Future Work

ORFormer demonstrates significant robustness in handling facial occlusions by combining a discrete codebook prior with a novel messenger token mechanism. While the model is highly effective against occlusions, the authors note it faces challenges with extreme facial deformations that deviate significantly from the learned codebook structures. Future work will explore extending ORFormer as a general-purpose feature extractor for other vision tasks, such as human pose estimation, where partial occlusion remains a major hurdle.

Summary updated

Abstract

Although facial landmark detection (FLD) has gained significant progress, existing FLD methods still suffer from performance drops on partially non-visible faces, such as faces with occlusions or under extreme lighting conditions or poses. To address this issue, we introduce ORFormer, a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically, ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way, the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings, enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features, ORFormer compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods, our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.
BibTeX
@inproceedings{chiang2025orformer,
  author = {Jui-Che Chiang and Hou-Ning Hu and Bo-Syuan Hou and Chia-Yu Tseng and Yu-Lun Liu and Min-Hung Chen and Yen-Yu Lin},
  title = {{ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=bBQyRDv982},
}

Loading related content...

Figure