SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation

Kejia Yin, Varshanth S. Rao, Ruowei Jiang, Xudong Liu, Parham Aarabi, David B. Lindell

Abstract

Facial landmark estimation is a critical task in computer vision, traditionally reliant on extensive human-labeled datasets. Self-supervised learning (SSL) offers a path toward label-free estimation, yet existing methods often struggle with fine-grained local details, high memory consumption, and noise from non-salient facial regions. We present SCE-MAE, a two-stage self-supervised framework that utilizes Masked Autoencoders (MAE) and a novel selective correspondence mechanism. By focusing on salient facial features and penalizing spatial ambiguity through Locality-Constrained Repellence (LCR) loss, SCE-MAE achieves state-of-the-art performance, reducing landmark matching errors by up to 44% while significantly improving computational efficiency.

Introduction

Facial landmark detection—the identification of specific coordinates such as eye corners, nose tips, and lip boundaries—is foundational for applications ranging from facial recognition to augmented reality. While supervised learning has dominated this field, the cost of manual annotation remains a significant bottleneck. Self-supervised learning (SSL) attempts to solve this by training models on unlabeled data using "pretext" tasks.

However, current SSL approaches for landmarks face three primary challenges:

  1. Scale Inconsistency: Many models use instance-level contrastive learning, which excels at distinguishing one face from another but fails to capture the fine-grained local geometry needed to distinguish specific landmarks.
  2. Computational Overhead: To achieve precision, many methods rely on "hypercolumns"—concatenating features from every layer of a neural network—which leads to excessive memory usage.
  3. Semantic Noise: Standard correspondence-based methods treat every pixel in an image with equal importance. This forces the model to learn relationships between non-informative regions (like cheeks or foreheads), diluting the focus on actual landmarks.

The SCE-MAE Framework

SCE-MAE addresses these limitations through a two-stage learning process designed to enhance local feature distinctiveness.

Stage 1: Region-level Pretraining with MAE

Instead of global image comparison, SCE-MAE begins with a Masked Autoencoder (MAE) pretext task. By masking random patches of a face and requiring the model to reconstruct the missing pixels, the network is forced to learn the underlying spatial structure and local geometry of facial features. This "mask-and-predict" approach provides a superior foundation for landmark tasks compared to traditional contrastive augmentation.

Stage 2: Selective Correspondence and CARB

To refine the features learned in Stage 1, the authors introduce the Correspondence Approximation and Refinement Block (CARB). This block leverages the model's internal attention mechanisms to separate "attentive" tokens (likely landmarks) from "inattentive" tokens (background or smooth skin).

To optimize memory, inattentive regions are clustered into a small set of representative points rather than being processed individually. The model then applies a novel Locality-Constrained Repellence (LCR) Loss. This loss function penalizes the model if it assigns similar feature signatures to points that are spatially distant. By enforcing this spatial prior, the model learns that the "left eye corner" must have a distinct signature from the "right eye corner," effectively eliminating semantic aliasing.

Experimental Results

Datasets and Benchmarking

The model was pretrained on the CelebA dataset (162,770 unlabeled images) and evaluated on standard benchmarks including MAFL, 300W, and AFLW. Notably, the authors identified significant labeling errors in the standard AFLW-R test set and released a corrected version, $AFLW_{RC}$, to ensure more accurate evaluation for future research.

Performance Metrics

SCE-MAE demonstrates substantial improvements over previous state-of-the-art (SOTA) methods like LEAD and ContrastLandmark:

  • Landmark Matching: On the MAFL dataset, SCE-MAE reduced Mean Pixel Error by 20% to 44%.
  • Landmark Detection: The framework achieved performance gains between 9% and 15% across various datasets.
  • Efficiency: Utilizing a smaller backbone (DeiT-T), SCE-MAE outperformed larger models that relied on memory-heavy hypercolumns, proving that selective attention is more effective than brute-force feature stacking.

Qualitative analysis via t-SNE visualizations confirms that SCE-MAE produces tighter, more distinct clusters for individual landmarks, successfully separating highly similar features like the left and right corners of the mouth.

Discussion and Limitations

While SCE-MAE sets a new benchmark for self-supervised landmark estimation, some challenges remain. To achieve high-resolution results, the model currently employs a "cover-and-stride" technique during inference, which requires multiple forward passes. This may limit its applicability in real-time mobile environments. Additionally, the method's reliance on the [CLS] token for region selection means that extreme occlusions or highly cluttered backgrounds could potentially interfere with the separation of attentive and inattentive tokens.

Conclusion

SCE-MAE represents a significant shift in self-supervised landmark estimation by prioritizing local spatial reconstruction and selective feature refinement. By moving away from dense, all-to-all pixel correspondence and embracing spatial priors through LCR loss, the framework provides a more accurate and computationally efficient solution for label-efficient facial analysis.

Summary updated

Abstract

Self-supervised landmark estimation is a challenging task that demands the formation of locally distinct feature representations to identify sparse facial landmarks in the absence of annotated data. To tackle this task, existing state-of-the-art (SOTA) methods (1) extract coarse features from backbones that are trained with instance-level self-supervised learning (SSL) paradigms, which neglect the dense prediction nature of the task, (2) aggregate them into memory-intensive hypercolumn formations, and (3) su-pervise lightweight projector networks to naï vely establish full local correspondences among all pairs of spatial features. In this paper, we introduce SCE-MAE, a framework that (1) leverages the MAE [14], a region-level SSL method that naturally better suits the landmark prediction task, (2) operates on the vanilla feature map instead of on expen-sive hypercolumns, and (3) employs a Correspondence Ap-proximation and Refinement Block (CARB) that utilizes a simple density peak clustering algorithm and our proposed Locality-Constrained Repellence Loss to directly hone only select local correspondences. We demonstrate through extensive experiments that SCE-MAE is highly effective and robust, outperforming existing SOTA methods by large mar-gins of ~20%-44% on the landmark matching and ~9%-15% on the landmark detection tasks.
BibTeX
@misc{yin2024sce,
  author = {Kejia Yin and Varshanth S. Rao and Ruowei Jiang and Xudong Liu and Parham Aarabi and David B. Lindell},
  title = {{SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation}},
  year = {2024},
  eprint = {2405.18322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.18322},
}
Figure