Abstract
Facial landmark estimation is a critical task in computer vision, traditionally reliant on extensive human-labeled datasets. Self-supervised learning (SSL) offers a path toward label-free estimation, yet existing methods often struggle with fine-grained local details, high memory consumption, and noise from non-salient facial regions. We present SCE-MAE, a two-stage self-supervised framework that utilizes Masked Autoencoders (MAE) and a novel selective correspondence mechanism. By focusing on salient facial features and penalizing spatial ambiguity through Locality-Constrained Repellence (LCR) loss, SCE-MAE achieves state-of-the-art performance, reducing landmark matching errors by up to 44% while significantly improving computational efficiency.
Introduction
Facial landmark detection—the identification of specific coordinates such as eye corners, nose tips, and lip boundaries—is foundational for applications ranging from facial recognition to augmented reality. While supervised learning has dominated this field, the cost of manual annotation remains a significant bottleneck. Self-supervised learning (SSL) attempts to solve this by training models on unlabeled data using "pretext" tasks.
However, current SSL approaches for landmarks face three primary challenges:
- Scale Inconsistency: Many models use instance-level contrastive learning, which excels at distinguishing one face from another but fails to capture the fine-grained local geometry needed to distinguish specific landmarks.
- Computational Overhead: To achieve precision, many methods rely on "hypercolumns"—concatenating features from every layer of a neural network—which leads to excessive memory usage.
- Semantic Noise: Standard correspondence-based methods treat every pixel in an image with equal importance. This forces the model to learn relationships between non-informative regions (like cheeks or foreheads), diluting the focus on actual landmarks.
The SCE-MAE Framework
SCE-MAE addresses these limitations through a two-stage learning process designed to enhance local feature distinctiveness.
Stage 1: Region-level Pretraining with MAE
Instead of global image comparison, SCE-MAE begins with a Masked Autoencoder (MAE) pretext task. By masking random patches of a face and requiring the model to reconstruct the missing pixels, the network is forced to learn the underlying spatial structure and local geometry of facial features. This "mask-and-predict" approach provides a superior foundation for landmark tasks compared to traditional contrastive augmentation.
Stage 2: Selective Correspondence and CARB
To refine the features learned in Stage 1, the authors introduce the Correspondence Approximation and Refinement Block (CARB). This block leverages the model's internal attention mechanisms to separate "attentive" tokens (likely landmarks) from "inattentive" tokens (background or smooth skin).
To optimize memory, inattentive regions are clustered into a small set of representative points rather than being processed individually. The model then applies a novel Locality-Constrained Repellence (LCR) Loss. This loss function penalizes the model if it assigns similar feature signatures to points that are spatially distant. By enforcing this spatial prior, the model learns that the "left eye corner" must have a distinct signature from the "right eye corner," effectively eliminating semantic aliasing.
Experimental Results
Datasets and Benchmarking
The model was pretrained on the CelebA dataset (162,770 unlabeled images) and evaluated on standard benchmarks including MAFL, 300W, and AFLW. Notably, the authors identified significant labeling errors in the standard AFLW-R test set and released a corrected version, $AFLW_{RC}$, to ensure more accurate evaluation for future research.
Performance Metrics
SCE-MAE demonstrates substantial improvements over previous state-of-the-art (SOTA) methods like LEAD and ContrastLandmark:
- Landmark Matching: On the MAFL dataset, SCE-MAE reduced Mean Pixel Error by 20% to 44%.
- Landmark Detection: The framework achieved performance gains between 9% and 15% across various datasets.
- Efficiency: Utilizing a smaller backbone (DeiT-T), SCE-MAE outperformed larger models that relied on memory-heavy hypercolumns, proving that selective attention is more effective than brute-force feature stacking.
Qualitative analysis via t-SNE visualizations confirms that SCE-MAE produces tighter, more distinct clusters for individual landmarks, successfully separating highly similar features like the left and right corners of the mouth.
Discussion and Limitations
While SCE-MAE sets a new benchmark for self-supervised landmark estimation, some challenges remain. To achieve high-resolution results, the model currently employs a "cover-and-stride" technique during inference, which requires multiple forward passes. This may limit its applicability in real-time mobile environments. Additionally, the method's reliance on the [CLS] token for region selection means that extreme occlusions or highly cluttered backgrounds could potentially interfere with the separation of attentive and inattentive tokens.
Conclusion
SCE-MAE represents a significant shift in self-supervised landmark estimation by prioritizing local spatial reconstruction and selective feature refinement. By moving away from dense, all-to-all pixel correspondence and embracing spatial priors through LCR loss, the framework provides a more accurate and computationally efficient solution for label-efficient facial analysis.
Summary updated