Detect Faces Efficiently: A Survey and Evaluations

Yuantao Feng, Shiqi Yu, Hanyang Peng, Yan-Ran Li, Jianguo Zhang

Abstract

Face detection is to search all the possible regions for faces in images and locate the faces if there are any. Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. In recent decades, researchers have created many typical and efficient face detectors from the Viola-Jones face detector to current CNN-based ones. However, with the tremendous increase in images and videos with variations in face scale, appearance, expression, occlusion and pose, traditional face detectors are challenged to detect various "in the wild" faces. The emergence of deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency. We further compare and discuss the popular and challenging datasets and their evaluation metrics. A comprehensive comparison of several successful deep learning-based face detectors is conducted to uncover their efficiency using two metrics: FLOPs and latency. The paper can guide to choose appropriate face detectors for different applications and also to develop more efficient and accurate detectors.

Introduction

In recent years, face detection accuracy has reached a point of near-saturation, with top models achieving over 0.92 Average Precision (AP) on challenging datasets like WIDER Face. However, this progress has come at the cost of massive computational overhead. Modern detectors must navigate a wide array of "in the wild" challenges, ranging from extreme scale variations to heavy occlusions and atypical poses.

Common challenges in face detection: (a) Simple case, (b) Scale, (c) Atypical pose, (d) Heavy occlusion, (e) Exaggerated expression, and (f) Extreme illumination.

The authors identify a major gap in how research is reported: many "state-of-the-art" results are achieved using multi-scale testing—feeding the same image into a network at multiple resolutions and merging the results. While this boosts accuracy, it multiplies the computational cost (FLOPs and latency) by several times, a fact often omitted in papers. This makes it difficult for engineers to choose the right model for real-world deployments on edge devices or real-time systems where power and speed are limited.

Architectural Frameworks

The evolution of deep learning-based face detection is organized into three main architectural frameworks:

  • Multi-stage: Inspired by traditional cascades (e.g., CascadeCNN), these use a series of networks to reject non-faces early in the pipeline.
  • Two-stage: Adapted from generic object detection (e.g., Faster R-CNN), these employ a Region Proposal Network (RPN) followed by a refinement stage.
  • One-stage: The current dominant trend (e.g., SSD, RetinaFace), performing feature extraction and detection in a single pass. These are further categorized into anchor-based (predefined boxes) and anchor-free (predicting center points or corners) methods.

Crucially, the authors propose a standardized FLOPs (Floating Point Operations) calculation method. They provide explicit mathematical formulas for every layer type (Convolution, Pooling, Batch Norm, etc.) to ensure that computational costs are compared on a level playing field, independent of specific hardware or software optimizations.

Benchmarks and Datasets

The survey reviews the primary benchmarks used to evaluate face detection performance:

  • WIDER Face: Currently the most popular and challenging dataset due to extreme variations in scale, occlusion, and pose.
  • FDDB & AFW: Older benchmarks where most modern detectors now achieve near-perfect scores.
  • Specialized Datasets: The paper also covers MAFA (masked faces), UFDD (extreme weather/blur), and DARK Face (low-light conditions), highlighting the move toward "in-the-wild" robustness.
Example of extreme scale variations in a crowd, a primary challenge in the WIDER Face dataset.

Critical Analysis: Accuracy vs. Efficiency

The paper provides several key insights into the trade-offs inherent in modern detection:

  • The Multi-Scale Trap: Adding more scales during testing provides diminishing returns. For example, in PyramidBox, moving from a single scale to a multi-scale ensemble increases FLOPs by over 3.5x while only providing a marginal boost in accuracy.
  • FLOPs vs. Latency: While FLOPs are a "neutral" measure of complexity, they don't always correlate linearly with latency. Factors like memory access patterns, hardware-specific optimizations, and the efficiency of post-processing (like Non-Maximum Suppression) play a significant role.
  • The "Comfort Zone": Every detector has a "comfort zone"—a range of face scales it was trained to handle. Multi-scale testing is essentially a brute-force way to move faces from the "out-of-zone" range back into the detector's comfort zone.
  • Anchor-Free Efficiency: Anchor-free methods (like CSP) are becoming highly competitive, as they avoid the computational burden of processing thousands of predefined anchor boxes.

Performance Evaluation

  • RetinaFace is highlighted as a standout performer, offering an excellent balance of high accuracy and relatively low computational cost compared to heavyweights like DSFD.
  • DSFD (Dual Shot Face Detector) is shown to have "unreasonable" computational costs—over 3x the FLOPs of similar models—for very small gains in AP.
  • Speed-Focusing Detectors: For edge applications, the authors evaluate lightweight models like ULFG (Ultra-Light-Fast-Generic-Face-Detector) and FaceBoxes. These models can run at over 100 FPS on a standard CPU while maintaining respectable accuracy for many practical applications.

Conclusion and Future Directions

The authors acknowledge several remaining frontiers in the field:

  • Superfast Detection: Achieving real-time 1080P video processing on low-cost edge devices remains a challenge, likely requiring models under 100M FLOPs.
  • The Long-Tailed Distribution: Most detectors struggle with "long-tail" variations, such as the elderly with deep wrinkles or faces with significant scarring.
  • Masked Face Detection: Robust detection of masked faces has become a fundamental necessity for modern systems in a post-pandemic world.

Summary updated

Abstract

Face detection is to search all the possible regions for faces in images and locate the faces if there are any. Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. In recent decades, researchers have created many typical and efficient face detectors from the Viola-Jones face detector to current CNN-based ones. However, with the tremendous increase in images and videos with variations in face scale, appearance, expression, occlusion and pose, traditional face detectors are challenged to detect various "in the wild" faces. The emergence of deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency. We further compare and discuss the popular and challenging datasets and their evaluation metrics. A comprehensive comparison of several successful deep learning-based face detectors is conducted to uncover their efficiency using two metrics: FLOPs and latency. The paper can guide to choose appropriate face detectors for different applications and also to develop more efficient and accurate detectors.
BibTeX
@article{feng2022detect,
  author = {Yuantao Feng and Shiqi Yu and Hanyang Peng and Yan-Ran Li and Jianguo Zhang},
  title = {{Detect Faces Efficiently: A Survey and Evaluations}},
  journal = {IEEE Transactions on Biometrics, Behavior, and Identity Science},
  year = {2022},
  url = {https://openreview.net/forum?id=DMSZUS6kTE},
}
Figure