Detect Faces Efficiently: A Survey and Evaluations

Yuantao Feng, Shiqi Yu, Hanyang Peng, Yan-Ran Li, Jianguo Zhang

Abstract

Face detection is to search all the possible regions for faces in images and locate the faces if there are any. Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. In recent decades, researchers have created many typical and efficient face detectors from the Viola-Jones face detector to current CNN-based ones. However, with the tremendous increase in images and videos with variations in face scale, appearance, expression, occlusion and pose, traditional face detectors are challenged to detect various “in the wild” faces. The emergence of deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency. We further compare and discuss the popular and challenging datasets and their evaluation metrics. A comprehensive comparison of several successful deep learning-based face detectors is conducted to uncover their efficiency using two metrics: FLOPs and latency. The paper can guide to choose appropriate face detectors for different applications and also to develop more efficient and accurate detectors.

The Efficiency Gap in Face Detection

While face detection is often considered a "solved" problem due to high Average Precision (AP) scores on benchmarks, these scores can be misleading. Many state-of-the-art models achieve top-tier accuracy by employing "multi-scale testing"—running the detector multiple times on different versions of the same image.

This approach boosts AP but causes a massive spike in computational cost, making such models impractical for real-time applications or edge deployment. There is a lack of standardized evaluation regarding the "price" of accuracy. This paper addresses that gap by providing a deep dive into the efficiency of various architectures and the impact of testing strategies on performance.

Architectural Evolution

The evolution of face detection can be categorized into three main frameworks:

  • Multi-stage (e.g., MTCNN): Uses a series of networks to propose, reject, and refine face candidates.
  • Two-stage (e.g., Faster R-CNN based): Uses a Region Proposal Network (RPN) followed by a dedicated classifier.
  • One-stage (e.g., RetinaFace, SSD-based): Performs feature extraction and bounding box regression in a single pass. These are currently the most popular for efficiency because their runtime is independent of the number of faces in an image.

A key contribution of this work is the derivation of standardized equations to calculate FLOPs (Floating Point Operations) for various neural network layers. The authors released a PyTorch-based tool to help researchers calculate these metrics consistently, moving away from hardware-dependent "latency" as the sole measure of speed.

Benchmarking "In the Wild"

The paper reviews several landmark datasets that test detectors against real-world variability.

  • WIDER Face: The current gold standard, featuring over 32,000 images with high variations in scale, occlusion, and pose.
  • FDDB & AFW: Older, smaller datasets used for initial benchmarking.
  • Specialized Datasets: Niche benchmarks like MAFA (masked faces), DARK Face (low-light), and 4K-Face (high-resolution).
Detecting faces with extreme variations in pose and occlusion.

The Accuracy-Efficiency Frontier

Extensive experiments were conducted to map the trade-off between precision and computational cost.

  • The Multi-Scale Trap: Some models use 10+ scales during testing to catch tiny faces, increasing computational load by an order of magnitude.
  • One-Stage Dominance: One-stage detectors like RetinaFace emerged as highly balanced performers, achieving top-tier AP with significantly lower FLOPs than competitors like DSFD.
  • Backbone Impact: Swapping heavy backbones (e.g., VGG-16) for MobileNet can drastically reduce FLOPs with marginal hits to accuracy.
  • Edge Performance: For resource-constrained devices, models like ULFG (Ultra-Light-Fast-Generic-Face-Detector) can achieve over 100 FPS on ARM processors, though they struggle with extreme conditions.
Face detection in a crowded scene, illustrating the challenge of varying scales.

Future Challenges

Despite progress, several challenges remain:

  • The Long Tail: Detecting faces in "long-tail" distributions—such as the elderly, people with facial injuries, or heavy makeup—remains difficult as datasets are often biased toward younger populations.
  • Superfast Requirements: There is no industry-standard for "superfast" detection. A true superfast detector should handle 1080p video in real-time on low-cost edge hardware with sub-100M FLOPs.
  • Masked Faces: While COVID-19 increased interest in masked face detection, integrated solutions are needed that don't require entirely separate models.

Summary updated

Abstract

Face detection is to search all the possible regions for faces in images and locate the faces if there are any. Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. In recent decades, researchers have created many typical and efficient face detectors from the Viola-Jones face detector to current CNN-based ones. However, with the tremendous increase in images and videos with variations in face scale, appearance, expression, occlusion and pose, traditional face detectors are challenged to detect various “in the wild” faces. The emergence of deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency. We further compare and discuss the popular and challenging datasets and their evaluation metrics. A comprehensive comparison of several successful deep learning-based face detectors is conducted to uncover their efficiency using two metrics: FLOPs and latency. The paper can guide to choose appropriate face detectors for different applications and also to develop more efficient and accurate detectors.
BibTeX
@article{feng2022detect,
  author = {Yuantao Feng and Shiqi Yu and Hanyang Peng and Yan-Ran Li and Jianguo Zhang},
  title = {{Detect Faces Efficiently: A Survey and Evaluations}},
  journal = {IEEE Transactions on Biometrics, Behavior, and Identity Science},
  year = {2022},
  url = {https://openreview.net/forum?id=DMSZUS6kTE},
}
Figure