TL;DR
This paper introduces TernaryFace, a novel, extremely low-bit (ternary, -1/0/1) face detector designed for "always-on" functionality directly on CMOS Image Sensors (CIS). By processing raw sensor data before the image signal processor (ISP) and employing quantization-aware training, TernaryFace achieves remarkable efficiency for mobile applications without sacrificing crucial performance.
What Problem Does This Paper Address?
Modern mobile devices and surveillance systems increasingly rely on "intelligent" CMOS Image Sensors (iCISs) that can perform on-sensor processing, particularly for always-on (AO) applications like face detection. However, existing deep learning-based face detectors are often too computationally heavy and designed for RGB images, failing to meet the stringent power, latency, and data format requirements of iCISs. Specifically, these detectors typically operate after the Image Signal Processor (ISP), which converts raw sensor data into RGB. This paper addresses the critical gap of developing efficient, AO face detectors that operate directly on raw sensor data before the ISP, minimizing power consumption and latency for always-on scenarios.
Methods
The authors propose a multi-pronged approach:
- Sensor-Aware Data Synthesis: They create a synthetic dataset (CelebCOCO) by combining CelebA and MS-COCO, emulating one-channel Bayer-patterned RAW sensor data. This process includes simulating sensor noise (Poisson and Gaussian), color degradation, and inverse gamma correction to mimic real-world CIS outputs. This is crucial for training models that understand raw sensor characteristics.
- Customized Lightweight Architecture: They adapt the RetinaFace [15] architecture, using a MobileNetV0.25 backbone. This architecture is significantly streamlined by reducing feature pyramid levels and simplifying the detection heads. They also limit channel counts to 128 to fit hardware constraints.
- Extreme Low-Bit Quantization: The core innovation is the use of ternary (-1, 0, 1) weights and 3-bit activations. They employ Quantization-Aware Training (QAT) to mitigate performance degradation. A key technique is Sharpness-aware Learned Step Size Quantization (SALSQ), which merges LSQ with Sharpness-Aware Minimization (SAM) to optimize quantizer step sizes for a flatter loss landscape.
- Batch Normalization Folding: To further boost efficiency, they fuse batch normalization layers with preceding convolution layers, reducing computational overhead.
Datasets
- CelebCOCO: A newly created synthetic dataset for training and evaluation, combining CelebA [51] and MS-COCO [49]. It emulates one-channel Bayer-patterned RAW sensor data and is designed to mimic AO FD scenarios for mobile CIS.
- WIDER FACE [71]: Used for additional experiments to evaluate the model's performance on a more challenging, standard face detection benchmark.
Key Insights and Ideas
- RAW Data is Key for AO: Processing face detection directly on RAW sensor data, before ISP conversion, is essential for achieving true "always-on" efficiency in mobile CIS applications. This bypasses significant processing steps and reduces data bandwidth.
- Ternary is Achievable and Efficient: The paper demonstrates that a ternary (-1/0/1) quantization scheme, combined with careful training and architecture design, can yield acceptable face detection performance while offering dramatic reductions in model size and computational cost.
- SALSQ Improves Quantization Robustness: Merging learned step size quantization with sharpness-aware minimization (SALSQ) proves effective in improving performance, especially for extremely low-bit quantization, by creating a more stable training landscape.
- Data Synthesis Matters: Creating realistic synthetic RAW data that accounts for sensor noise and processing artifacts is critical for training models that perform well in real-world CIS environments.
Key Results
- Efficiency: TernaryFace achieves a significantly smaller model size (27M parameters) and lower computational cost (0.012 GFLOPS, 27M BOPs) compared to full-precision RetinaFace (2168M parameters, 0.423 GFLOPS, 2168M BOPs) and even its own 4-bit and 8-bit quantized versions.
- Performance: On the synthetic CelebCOCO dataset, TernaryFace achieves a competitive [email protected] of 95.1%, with a false positive rate of 2.47%. While performance drops on the more challenging WIDER FACE dataset ([email protected] of 32.9% on the Hard subset), it still demonstrates the viability of the approach.
- Comparison: TernaryFace significantly outperforms Viola-Jones [66] in terms of performance and efficiency for AO FD applications.
Limitations and Future Work
- WIDER FACE Performance: The authors acknowledge that TernaryFace's performance on the "Hard" subset of WIDER FACE (which contains many tiny faces) is lower. This is attributed to the model's design prioritizing larger frontal faces for the AO FD scenario and the challenges of tiny face detection.
- Generalizability to Diverse CIS: The performance might vary across different manufacturers' CIS sensors due to variations in RAW data characteristics. Further validation on real-world data from various manufacturers is needed.
- Future Work: The paper suggests that further research into hardware-software co-design and comprehensive HW design validation is necessary for on-sensor implementation. Exploring more advanced quantization techniques and architectures tailored for specific CIS constraints could also be beneficial.
Summary updated