Beyond Trade-off: Accelerate FCN-based Face Detector with Higher Accuracy
Modern face detection relies on Fully Convolutional Networks (FCNs) which, while accurate, suffer from high computational costs due to exhaustive search across image pyramids. This paper introduces $S^2AP$ (Scale Estimation and Spatial Attention Proposal), a lightweight mechanism that predicts relevant scales and spatial locations to prune redundant computations. By employing masked convolutions, the proposed method achieves significant speedups—up to 4x for standard detectors—while simultaneously improving state-of-the-art accuracy by filtering background noise.
Introduction: The Efficiency Challenge in Face Detection
Fully Convolutional Networks (FCNs) are the backbone of modern face detection. However, to detect faces of varying sizes, these models typically process an "image pyramid"—resizing the input image multiple times and running the detector on every version. This approach introduces two major inefficiencies:
- Spatial Redundancy: Most of an image consists of background, yet standard networks convolve every pixel with equal effort.
- Scale Redundancy: A face typically exists at only a few scales within the pyramid, but detectors often process 20-60 different scales.
While "single-shot" detectors (e.g., SSD, YOLO) attempt to mitigate this by trading accuracy for speed, this research explores whether the search space can be pruned to compute only what is necessary without sacrificing precision.
Proposed Method: $S^2AP$ and Masked Convolutions
The authors propose a two-stage pipeline designed to identify "where" and "at what scale" a face exists before performing heavy computation.
- $S^2AP$ (The Attention Model): A lightweight scout model (based on a shallow ResNet18) takes a single resized image and outputs a 3D attention map. Each of the 60 channels represents a specific scale range. An active "pixel" in a channel indicates a high probability of a face at that specific size and location.
- Soft Ground Truth: To ensure robustness against noise and slight misalignments, the model is trained using "soft" labels, where neighboring scales and locations in the attention map receive partial activation.
- Masked-Convolution: This is the primary engine for acceleration. Using the $S^2AP$ map as a mask, standard dense convolutions are converted into sparse matrix multiplications. Computation is only performed for "active" regions identified by the scout.
- SSCU (Scale-Spatial Compute Unit): This unit aggregates attention results to guide the final detector (e.g., a Region Proposal Network), ensuring it focuses only on relevant portions of the image pyramid.
Key Results and Insights
The $S^2AP$ method demonstrates that high-level guidance can significantly optimize low-level operations. Because the "scout" model is extremely shallow, its overhead is negligible compared to the computational savings achieved through masking.
- Significant Speedup: The method accelerated the baseline RPN by approximately 4x on average while maintaining a 98% recall rate. On an NVIDIA P100 GPU, inference latency dropped from ~95ms to between 14ms and 28ms.
- Improved Accuracy: By forcing the network to ignore "invalid" regions, the attention mechanism acts as a filter for false positives. The model achieved an AP of 99.94% on AFW and improved the True Positive Rate on FDDB from 91.39% to 93.59% at a fixed false positive count.
- Generality: High recall on COCO and VOC suggests that the "scale + spatial" attention strategy is effective for generic object detection beyond faces.
Limitations and Future Directions
The system's performance is intrinsically tied to the accuracy of the $S^2AP$ scout; a false negative at this stage prevents the main detector from finding the face. Furthermore, while theoretical speedups are high, realizing these gains on standard hardware requires specialized software implementations, such as custom CUDA kernels for sparse operations. Finally, while effective for faces, objects with highly varied aspect ratios might require more complex attention map configurations.
Summary updated