Lightweight Real-time Makeup Try-on in Mobile Browsers with Tiny CNN Models for Facial Tracking

Tianxing Li, Zhi Yu, Edmund Phung, Brendan Duke, Irina Kezele, Parham AarabiarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel, lightweight two-stage CNN architecture for highly accurate facial landmark detection that runs efficiently in mobile web browsers. This enables real-time virtual makeup try-on experiences directly on smartphones without requiring dedicated apps, addressing the computational and size constraints of such devices.

What Problem Does This Paper Address?

Accurate facial alignment is crucial for realistic virtual makeup try-on applications. While deep learning models have achieved impressive accuracy, they are often too large and computationally demanding for resource-constrained mobile devices and web environments. This leads to slow loading times and poor user experiences. The authors aim to bridge this gap by developing a compact, fast, and accurate facial tracking model suitable for client-side execution in mobile web browsers.

Methods

The core of the paper is a two-stage CNN architecture designed for efficiency and accuracy.

  1. Coarse-to-Fine Localization: The model first makes a coarse prediction of facial landmarks. Then, it uses these initial predictions to extract "Regions of Interest" (RoIs) from shared convolutional features. These RoIs are fed into a second stage for more spatially refined landmark predictions.
  2. Efficient Architecture: The architecture leverages inverted residual blocks (inspired by MobileNetV2) which significantly reduce computation and model size by using depthwise separable convolutions.
  3. Heatmap Regression with RoI Align: Instead of high-resolution heatmaps, the model uses low-resolution heatmaps for landmark prediction. The RoI align technique (borrowed from Mask R-CNN) is used to efficiently extract features for each landmark's RoI, avoiding redundant computations.
  4. Loss Function: A combination of pixelwise sigmoid cross-entropy loss for heatmaps and an L2 distance loss is used to train the model. The L2 loss helps mitigate issues with heatmaps being cut off near image boundaries.
  5. Rendering Pipeline: After landmark detection, a separate rendering pipeline is used to apply makeup effects, including lighting estimation, mask generation, and applying color, gloss, and wetness.

Datasets

The authors created a custom dataset by combining:

  • Helen [11]
  • LFPW [1]
  • iBUG [17]
  • Additional in-house collected images.

This combined dataset contains 3681 images with 65 landmarks (62 inner points and 3 contour points). They also evaluated their model on the 300W dataset [16] for comparison with state-of-the-art methods.

Key Insights and Ideas

  • Coarse-to-Fine is Key for Efficiency: The two-stage approach, where an initial coarse prediction guides a more refined second stage, allows the model to focus computation on relevant areas, significantly improving efficiency.
  • RoI Align for Landmark Specificity: Applying RoI align to extract features for each landmark's RoI is crucial. It allows the model to learn more specific features for each landmark and avoids recomputing features for overlapping regions.
  • Low-Resolution Heatmaps are Sufficient: The paper demonstrates that even with low-resolution heatmaps, high accuracy can be achieved, especially when combined with the two-stage approach and RoI align. This drastically reduces computational load.
  • Balancing Size and Accuracy: The use of efficient building blocks like inverted residual blocks and careful architectural design allows for a significant reduction in model size (down to 607KB for their demo model) without a substantial drop in accuracy.

Key Results

  • Compact Model Size: Their demo model achieved a size of 607KB, making it ideal for web deployment.
  • Fast Inference: The model achieved an inference time of 20ms on an iPhone XR.
  • Competitive Accuracy: The proposed model achieved competitive accuracy on the 300W dataset, with an "Inner Error" of 3.21 and "Contour Error" of 9.00 for their full model (α=1). Even a half-sized model (α=0.5) maintained similar accuracy.
  • Ablation Study: An ablation study confirmed that all components (heatmap loss, L2 loss, and the second stage) contribute significantly to the overall performance.

Limitations and Future Work

  • Pose Distribution: The authors note that their custom dataset has a pose distribution of approximately -30° to 30° in yaw angle. While this aligns with their application scenario, it might limit generalization to extreme poses.
  • 300W Full Dataset Drop: A slight drop in accuracy on the 300W Full dataset compared to LAB is attributed to their architectural design being more tailored for unoccluded, mostly frontal faces.
  • Future Work: The paper implicitly suggests that further model size reduction and speed optimization are possible, as demonstrated by their α=0.5 model. The focus on real-time performance in mobile browsers opens avenues for more sophisticated AR/VR applications directly on devices.

Summary updated

Abstract

Recent works on convolutional neural networks (CNNs) for facial alignment have demonstrated unprecedented accuracy on a variety of large, publicly available datasets. However, the developed models are often both cumbersome and computationally expensive, and are not adapted to applications on resource restricted devices. In this work, we look into developing and training compact facial alignment models that feature fast inference speed and small deployment size, making them suitable for applications on the aforementioned category of devices. Our main contribution lies in designing such small models while maintaining high accuracy of facial alignment. The models we propose make use of light CNN architectures adapted to the facial alignment problem for accurate two-stage prediction of facial landmark coordinates from low-resolution output heatmaps. We further combine the developed facial tracker with a rendering method, and build a real-time makeup try-on demo that runs client-side in smartphone Web browsers. More results and demo are in our project page: http://research.modiface.com/makeup-try-on-cvprw2019/
BibTeX
@misc{li2019lightweight,
  author = {Tianxing Li and Zhi Yu and Edmund Phung and Brendan Duke and Irina Kezele and Parham Aarabi},
  title = {{Lightweight Real-time Makeup Try-on in Mobile Browsers with Tiny CNN Models for Facial Tracking}},
  year = {2019},
  eprint = {1906.02260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.02260},
}
Figure