RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection

Ashish Kumar, Prashant Kumar, Mallika GargarXiv 2026 · arXiv

TL;DR

RCFormer is a novel transformer-based framework that significantly improves facial landmark detection in "in-the-wild" scenarios where faces are partially hidden by occlusions, poor lighting, or extreme poses. By explicitly modeling which parts of a face are reliable and using that information to "hallucinate" missing regions from context, it achieves state-of-the-art accuracy on the most challenging industry benchmarks.

What Problem Does This Paper Address?

Facial landmark detection (FLD)—identifying key points like the corners of eyes, nose tip, and jawline—is a solved problem for clear, front-facing photos. However, it breaks down in the real world. When a person wears sunglasses, covers their mouth with a hand, or turns their head sharply, the "local evidence" for those landmarks disappears.

Previous attempts to solve this used "context-guided recovery," essentially trying to guess where an occluded eye is by looking at the rest of the face. But these methods had a major flaw: they often suffered from a "garbage in, garbage out" problem. If the surrounding context was also noisy or occluded, the model would propagate that bad information, leading to "patchy" or anatomically impossible results. RCFormer addresses this by making the recovery process reliability-aware.

Methods

The authors propose a two-stage approach:

  1. Learning a Quantized Heatmap Prior: Before tackling occlusions, the model is trained on clean, perfect face images to learn what a "valid" face looks like. It uses a VQVAE-style architecture to create a "codebook" (a dictionary) of plausible facial edge patterns. The quantized heatmap prior learning stage
Figure 1: Overview of the quantized heatmap prior learning stage. A clean face image is encoded into latent features, quantized through a learned codebook, and decoded into an edge heatmap representation.
  1. RCFormer Recovery: This is the core innovation. It processes an image in patches and maintains two parallel streams of information: a Regular stream (what is actually visible) and a Context stream (what is inferred). Three new components make this work:
  2. Visible-Context Conditioning (VCC): It creates a global summary of the face, weighted by visibility, to ensure the inferred parts match the specific identity and pose of the input face.
  3. Visibility-Guided Context Aggregation (VGCA): During the "attention" phase, the model is told to ignore patches it has flagged as unreliable, preventing corrupted data from "bleeding" into the recovered features.
  4. Spatial Residual Correction (SRC): A final refinement step that ensures the recovered patches blend smoothly with their neighbors, maintaining the natural continuity of facial contours.

The final output is a high-fidelity "recovered heatmap" that guides a standard landmark detector to the correct coordinates. RCFormer Architecture Overview

RCFormer Architecture Overview

Datasets

The model was evaluated on three gold-standard benchmarks:

  • WFLW (Wider Facial Landmarks in-the-wild): 10,000 images with 98 landmarks, featuring heavy occlusions, blur, and makeup.
  • COFW (Caltech Occluded Faces in the Wild): Specifically designed to test robustness against heavy facial occlusion (29 landmarks).
  • 300W: A standard benchmark (68 landmarks) split into "common" and "challenging" subsets.

Key Insights and Ideas

  • Recursive Reliability: The "aha moment" here is that you can't just use context to fix occlusion; you first have to know which context is trustworthy. By weighting the attention mechanism based on a predicted "unreliability map," the model effectively filters its own input.
  • Discrepancy as a Signal: The model identifies occlusions by measuring the "disagreement" between what it sees locally and what it expects to see based on the rest of the face. A high discrepancy is a strong signal that a patch is occluded.
  • Structural Guidance: Instead of trying to predict coordinates directly from a messy image, RCFormer focuses on recovering a clean structural map (edges). This structural cue is much easier for downstream networks to interpret than raw, occluded pixels.

Key Results

RCFormer set new state-of-the-art (SOTA) records across all three datasets:

  • Accuracy: On the WFLW dataset, it achieved a Normalized Mean Error (NME) of 3.73, outperforming the previous best (ORFormer) and significantly beating standard backbones like HRNet.
  • Robustness: On the COFW dataset (heavy occlusion), it achieved an NME of 4.29, a substantial improvement over previous methods.
  • Heatmap Quality: Quantitatively, RCFormer reduced the error in recovered heatmaps by 24% to 41% compared to baselines. Qualitatively, it produces much cleaner facial structures, especially around occluded boundaries. Qualitative comparison showing RCFormer's superior accuracy on occluded faces
Qualitative comparison showing RCFormer's superior accuracy on occluded faces

Limitations and Future Work

While highly effective, the authors note that the system still struggles when occlusions are extremely large (e.g., more than half the face is covered), as there isn't enough contextual evidence left to reconstruct the missing parts.

Future work includes testing the model's cross-dataset generalization—seeing how a model trained on one dataset performs on entirely different data distributions—and further optimizing the computational overhead for real-time mobile applications.

Summary updated

Abstract

Facial landmark detection has advanced substantially under regular imaging conditions, yet occlusion, poor illumination, and large pose changes remain major sources of error because local facial evidence can become incomplete or misleading. Context-guided latent patch recovery offers a promising way to address this issue by inferring unreliable patch representations from surrounding facial context before heatmap prediction. However, the recovered context can itself be unreliable when it is weakly adapted to the input face, influenced by corrupted neighboring patches, or blended without preserving local spatial coherence. We propose RCFormer, a Reliability-aware Context Transformer for occlusion-robust facial landmark detection. RCFormer strengthens latent patch recovery with three reliability-oriented components: visible-context conditioning (VCC), which adapts the context representation using a visibility-weighted global facial summary; visibility-guided context aggregation (VGCA), which reduces the influence of unreliable source patches during attention; and spatial residual correction (SRC), which refines blended latent features using neighboring patch information. The recovered heatmaps are then used as reliability-aware structural cues for existing landmark detection networks. Experiments show that RCFormer improves heatmap recovery and achieves strong landmark localization results on the WFLW, COFW, and 300W datasets.
BibTeX
@misc{kumar2026rcformer,
  author = {Ashish Kumar and Prashant Kumar and Mallika Garg},
  title = {{RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection}},
  year = {2026},
  eprint = {2611.01822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01822},
}
Figure