Towards Flops-constrained Face Recognition

Yu Liu, Guanglu Song, Manyuan Zhang, Jihao Liu, Yucong Zhou, Junjie Yan

TL;DR

This paper presents the winning strategies of team 'trojans' from the ICCV19 Lightweight Face Recognition Challenge, which imposed a strict 30 GFlops computational limit. The authors introduce Efficient PolyFace (a custom-searched architecture), ArcNegFace (a loss function designed for hard negative mining), and QAN++ (an advanced video frame aggregation method). Together, these innovations achieved state-of-the-art performance across both image and video face recognition tracks.

What Problem Does This Paper Address?

While deep learning has pushed face recognition accuracy to remarkable heights, the top-performing models are typically massive and computationally expensive. Deploying these models on edge devices or in large-scale video processing pipelines requires strict efficiency.

The ICCV19 Lightweight Face Recognition Challenge formalized this trade-off by restricting submissions to a single model with a computational budget of no higher than 30 GFlops. The core challenge is maximizing accuracy—maintaining robustness against noise, blur, and pose variations—while strictly adhering to this computational ceiling.

Methods

The authors tackle the FLOPs constraint through a three-pronged approach spanning architecture search, loss function design, and video frame aggregation:

  1. Efficient PolyFace Architecture: Starting with a baseline architecture called PolyFace (inspired by PolyNet), which repeats basic blocks and utilizes a "stem-enrichment block" to upscale and downscale spatial sizes, the authors launch a Neural Architecture Search (NAS). Inspired by EfficientNet, this search optimizes the network's depth and width to squeeze maximum performance out of the 30 GFlops budget.
Figure 1. The details of blocks in PolyFace. The numbers in block Conv+BN +ReLU represent the input channel, output channel, kernel size, stride, and padding. The numbers in block Max Pooling represent the kernel size and stride. The numbers in block Conv + BN represent the input channel, output channel, kernel size, stride, and padding.
  1. ArcNegFace Loss Function: Standard ArcFace enforces an angular margin between embeddings and class anchors but treats all negative classes equally. ArcNegFace improves upon this by considering the distance between anchors. It uses a Gaussian function to adaptively weight negative logits, effectively performing hard negative mining and reducing the disruptive impact of incorrectly labeled training data.
  2. QAN++ (Enhanced Quality Aware Network): For video-based recognition, instead of treating all frames equally via simple average pooling, the authors introduce a lightweight quality generation branch. QAN++ estimates a discriminative quality score for each frame based on its feature clarity relative to class anchors. During inference, these scores are dynamically re-scaled and used to compute a weighted sum of frame representations.
  3. Bag of Tricks: The implementation is bolstered by several optimization tweaks, including anchor finetuning (re-initializing classification layer weights with the mean features of training identities before final finetuning), scale/shift augmentations, color jitter, and AdaBN to mitigate domain shift between training and testing sets.

Datasets

The models were trained and evaluated using the challenge's strictly mandated datasets:

  • DeepGlint-Large: Used for the large-scale image-based face recognition track.
  • iQiyi-Large: Used for the large-scale video-based face recognition track.

To ensure fairness, the challenge required all participants to use identical training data, face detectors, and alignment methods.

Key Insights and Ideas

  • Custom Scaling Beats Standard Backbones: Standard architectures like ResNet-100 are not inherently optimized for arbitrary FLOP ceilings. The authors show that compound NAS scaling of width and depth yields significantly better accuracy under a strict computational budget.
  • Adaptive Negative Margins Matter: Incorporating anchor-to-anchor distances into the loss function allows the model to focus on genuinely confusing identities (hard negatives) while ignoring label noise, leading to cleaner embedding spaces.
  • Relative Quality in Video Aggregation: In video tracks, absolute frame quality matters less than relative quality within the same clip. QAN++ dynamically re-scales frame scores based on the maximum and minimum quality found within that specific video sequence, preventing a few mediocre frames from ruining a high-quality aggregate representation.

Key Results

  • Video Track (iQiyi-Large): The PolyFace + QAN++ solution (consuming 24.12 GFlops) achieved a True Positive Rate (TPR) of 72.981% @ FPR=1e-4, winning 1st place in the competition. This significantly outperformed standard average pooling (65.843%).
  • Image Track (DeepGlint-Large): The Efficient PolyFace model (consuming 28.25 GFlops) achieved a TPR of 93.801% @ FPR=1e-8, which improved to 94.198% when adjusted with AdaBN, securing 2nd place on the leaderboard.
  • Loss Function Impact: Replacing standard ArcFace with ArcNegFace on a baseline PolyNet model boosted the TPR from 90.829% to 91.639% under identical training configurations.

Limitations and Future Work

The authors note that while certain regularization techniques like RegularFace (which constrains inter-class distance) were explored, they consumed excessive memory and yielded negligible performance gains.

Additionally, the top image-track result relied on AdaBN, which utilizes image-level information from the test set to minimize domain shift. For scenarios where test-set distribution is entirely unknown or streaming, the model's performance drops slightly to its base accuracy of 93.801%. Future work points toward developing more memory-efficient regularization methods and completely blind domain adaptation techniques.

Summary updated

Abstract

Large scale face recognition is challenging especially when the computational budget is limited. Given a \textit{flops} upper bound, the key is to find the optimal neural network architecture and optimization method. In this article, we briefly introduce the solutions of team 'trojans' for the ICCV19 - Lightweight Face Recognition Challenge~\cite{lfr}. The challenge requires each submission to be one single model with the computational budget no higher than 30 GFlops. We introduce a searched network architecture `Efficient PolyFace' based on the Flops constraint, a novel loss function `ArcNegFace', a novel frame aggregation method `QAN++', together with a bag of useful tricks in our implementation (augmentations, regular face, label smoothing, anchor finetuning, etc.). Our basic model, `Efficient PolyFace', takes 28.25 Gflops for the `deepglint-large' image-based track, and the `PolyFace+QAN++' solution takes 24.12 Gflops for the `iQiyi-large' video-based track. These two solutions achieve 94.198\% @ 1e-8 and 72.981\% @ 1e-4 in the two tracks respectively, which are the state-of-the-art results.
BibTeX
@misc{liu2019towards,
  author = {Yu Liu and Guanglu Song and Manyuan Zhang and Jihao Liu and Yucong Zhou and Junjie Yan},
  title = {{Towards Flops-constrained Face Recognition}},
  year = {2019},
  eprint = {1909.00632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.00632},
}
Figure