TL;DR
This paper introduces a novel, lightweight two-stage CNN architecture for highly accurate facial landmark detection that runs efficiently in mobile web browsers. This enables real-time virtual makeup try-on experiences directly on smartphones without requiring dedicated apps, addressing the computational and size constraints of such devices.
What Problem Does This Paper Address?
Accurate facial alignment is crucial for realistic virtual makeup try-on applications. While deep learning models have achieved impressive accuracy, they are often too large and computationally demanding for resource-constrained mobile devices and web environments. This leads to slow loading times and poor user experiences. The authors aim to bridge this gap by developing a compact, fast, and accurate facial tracking model suitable for client-side execution in mobile web browsers.
Methods
The core of the paper is a two-stage CNN architecture designed for efficiency and accuracy.
- Coarse-to-Fine Localization: The model first makes a coarse prediction of facial landmarks. Then, it uses these initial predictions to extract "Regions of Interest" (RoIs) from shared convolutional features. These RoIs are fed into a second stage for more spatially refined landmark predictions.
- Efficient Architecture: The architecture leverages inverted residual blocks (inspired by MobileNetV2) which significantly reduce computation and model size by using depthwise separable convolutions.
- Heatmap Regression with RoI Align: Instead of high-resolution heatmaps, the model uses low-resolution heatmaps for landmark prediction. The RoI align technique (borrowed from Mask R-CNN) is used to efficiently extract features for each landmark's RoI, avoiding redundant computations.
- Loss Function: A combination of pixelwise sigmoid cross-entropy loss for heatmaps and an L2 distance loss is used to train the model. The L2 loss helps mitigate issues with heatmaps being cut off near image boundaries.
- Rendering Pipeline: After landmark detection, a separate rendering pipeline is used to apply makeup effects, including lighting estimation, mask generation, and applying color, gloss, and wetness.
Datasets
The authors created a custom dataset by combining:
- Helen [11]
- LFPW [1]
- iBUG [17]
- Additional in-house collected images.
This combined dataset contains 3681 images with 65 landmarks (62 inner points and 3 contour points). They also evaluated their model on the 300W dataset [16] for comparison with state-of-the-art methods.
Key Insights and Ideas
- Coarse-to-Fine is Key for Efficiency: The two-stage approach, where an initial coarse prediction guides a more refined second stage, allows the model to focus computation on relevant areas, significantly improving efficiency.
- RoI Align for Landmark Specificity: Applying RoI align to extract features for each landmark's RoI is crucial. It allows the model to learn more specific features for each landmark and avoids recomputing features for overlapping regions.
- Low-Resolution Heatmaps are Sufficient: The paper demonstrates that even with low-resolution heatmaps, high accuracy can be achieved, especially when combined with the two-stage approach and RoI align. This drastically reduces computational load.
- Balancing Size and Accuracy: The use of efficient building blocks like inverted residual blocks and careful architectural design allows for a significant reduction in model size (down to 607KB for their demo model) without a substantial drop in accuracy.
Key Results
- Compact Model Size: Their demo model achieved a size of 607KB, making it ideal for web deployment.
- Fast Inference: The model achieved an inference time of 20ms on an iPhone XR.
- Competitive Accuracy: The proposed model achieved competitive accuracy on the 300W dataset, with an "Inner Error" of 3.21 and "Contour Error" of 9.00 for their full model (α=1). Even a half-sized model (α=0.5) maintained similar accuracy.
- Ablation Study: An ablation study confirmed that all components (heatmap loss, L2 loss, and the second stage) contribute significantly to the overall performance.
Limitations and Future Work
- Pose Distribution: The authors note that their custom dataset has a pose distribution of approximately -30° to 30° in yaw angle. While this aligns with their application scenario, it might limit generalization to extreme poses.
- 300W Full Dataset Drop: A slight drop in accuracy on the 300W Full dataset compared to LAB is attributed to their architectural design being more tailored for unoccluded, mostly frontal faces.
- Future Work: The paper implicitly suggests that further model size reduction and speed optimization are possible, as demonstrated by their α=0.5 model. The focus on real-time performance in mobile browsers opens avenues for more sophisticated AR/VR applications directly on devices.
Summary updated