Fully Quantized Always-on Face Detector Considering Mobile Image Sensors

Haechang Lee, Wongi Jeong, Dongil Ryu, Hyunwoo Je, Albert No, Kijeong Kim, Se Young ChunarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces TernaryFace, a novel, extremely low-bit (ternary, -1/0/1) face detector designed for "always-on" functionality directly on CMOS Image Sensors (CIS). By processing raw sensor data before the image signal processor (ISP) and employing quantization-aware training, TernaryFace achieves remarkable efficiency for mobile applications without sacrificing crucial performance.

What Problem Does This Paper Address?

Modern mobile devices and surveillance systems increasingly rely on "intelligent" CMOS Image Sensors (iCISs) that can perform on-sensor processing, particularly for always-on (AO) applications like face detection. However, existing deep learning-based face detectors are often too computationally heavy and designed for RGB images, failing to meet the stringent power, latency, and data format requirements of iCISs. Specifically, these detectors typically operate after the Image Signal Processor (ISP), which converts raw sensor data into RGB. This paper addresses the critical gap of developing efficient, AO face detectors that operate directly on raw sensor data before the ISP, minimizing power consumption and latency for always-on scenarios.

Methods

The authors propose a multi-pronged approach:

  • Sensor-Aware Data Synthesis: They create a synthetic dataset (CelebCOCO) by combining CelebA and MS-COCO, emulating one-channel Bayer-patterned RAW sensor data. This process includes simulating sensor noise (Poisson and Gaussian), color degradation, and inverse gamma correction to mimic real-world CIS outputs. This is crucial for training models that understand raw sensor characteristics.
  • Customized Lightweight Architecture: They adapt the RetinaFace [15] architecture, using a MobileNetV0.25 backbone. This architecture is significantly streamlined by reducing feature pyramid levels and simplifying the detection heads. They also limit channel counts to 128 to fit hardware constraints.
  • Extreme Low-Bit Quantization: The core innovation is the use of ternary (-1, 0, 1) weights and 3-bit activations. They employ Quantization-Aware Training (QAT) to mitigate performance degradation. A key technique is Sharpness-aware Learned Step Size Quantization (SALSQ), which merges LSQ with Sharpness-Aware Minimization (SAM) to optimize quantizer step sizes for a flatter loss landscape.
  • Batch Normalization Folding: To further boost efficiency, they fuse batch normalization layers with preceding convolution layers, reducing computational overhead.

Datasets

  • CelebCOCO: A newly created synthetic dataset for training and evaluation, combining CelebA [51] and MS-COCO [49]. It emulates one-channel Bayer-patterned RAW sensor data and is designed to mimic AO FD scenarios for mobile CIS.
  • WIDER FACE [71]: Used for additional experiments to evaluate the model's performance on a more challenging, standard face detection benchmark.

Key Insights and Ideas

  • RAW Data is Key for AO: Processing face detection directly on RAW sensor data, before ISP conversion, is essential for achieving true "always-on" efficiency in mobile CIS applications. This bypasses significant processing steps and reduces data bandwidth.
  • Ternary is Achievable and Efficient: The paper demonstrates that a ternary (-1/0/1) quantization scheme, combined with careful training and architecture design, can yield acceptable face detection performance while offering dramatic reductions in model size and computational cost.
  • SALSQ Improves Quantization Robustness: Merging learned step size quantization with sharpness-aware minimization (SALSQ) proves effective in improving performance, especially for extremely low-bit quantization, by creating a more stable training landscape.
  • Data Synthesis Matters: Creating realistic synthetic RAW data that accounts for sensor noise and processing artifacts is critical for training models that perform well in real-world CIS environments.

Key Results

  • Efficiency: TernaryFace achieves a significantly smaller model size (27M parameters) and lower computational cost (0.012 GFLOPS, 27M BOPs) compared to full-precision RetinaFace (2168M parameters, 0.423 GFLOPS, 2168M BOPs) and even its own 4-bit and 8-bit quantized versions.
  • Performance: On the synthetic CelebCOCO dataset, TernaryFace achieves a competitive [email protected] of 95.1%, with a false positive rate of 2.47%. While performance drops on the more challenging WIDER FACE dataset ([email protected] of 32.9% on the Hard subset), it still demonstrates the viability of the approach.
  • Comparison: TernaryFace significantly outperforms Viola-Jones [66] in terms of performance and efficiency for AO FD applications.

Limitations and Future Work

  • WIDER FACE Performance: The authors acknowledge that TernaryFace's performance on the "Hard" subset of WIDER FACE (which contains many tiny faces) is lower. This is attributed to the model's design prioritizing larger frontal faces for the AO FD scenario and the challenges of tiny face detection.
  • Generalizability to Diverse CIS: The performance might vary across different manufacturers' CIS sensors due to variations in RAW data characteristics. Further validation on real-world data from various manufacturers is needed.
  • Future Work: The paper suggests that further research into hardware-software co-design and comprehensive HW design validation is necessary for on-sensor implementation. Exploring more advanced quantization techniques and architectures tailored for specific CIS constraints could also be beneficial.

Summary updated

Abstract

Despite significant research on lightweight deep neural networks (DNNs) designed for edge devices, the current face detectors do not fully meet the requirements for "intelligent" CMOS image sensors (iCISs) integrated with embedded DNNs. These sensors are essential in various practical applications, such as energy-efficient mobile phones and surveillance systems with always-on capabilities. One noteworthy limitation is the absence of suitable face detectors for the always-on scenario, a crucial aspect of image sensor-level applications. These detectors must operate directly with sensor RAW data before the image signal processor (ISP) takes over. This gap poses a significant challenge in achieving optimal performance in such scenarios. Further research and development are necessary to bridge this gap and fully leverage the potential of iCIS applications. In this study, we aim to bridge the gap by exploring extremely low-bit lightweight face detectors, focusing on the always-on face detection scenario for mobile image sensor applications. To achieve this, our proposed model utilizes sensor-aware synthetic RAW inputs, simulating always-on face detection processed "before" the ISP chain. Our approach employs ternary (-1, 0, 1) weights for potential implementations in image sensors, resulting in a relatively simple network architecture with shallow layers and extremely low-bitwidth. Our method demonstrates reasonable face detection performance and excellent efficiency in simulation studies, offering promising possibilities for practical always-on face detectors in real-world applications.
BibTeX
@misc{lee2023fully,
  author = {Haechang Lee and Wongi Jeong and Dongil Ryu and Hyunwoo Je and Albert No and Kijeong Kim and Se Young Chun},
  title = {{Fully Quantized Always-on Face Detector Considering Mobile Image Sensors}},
  year = {2023},
  eprint = {2311.01001},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.01001},
}
Figure