Peer Review #1 of "Efficient video face recognition based on frame selection and quality assessment (v0.1)"

Angelina Kharchevnikova, A. Savchenko, F Alonso-Fernandez, J Fierrez, J Ortega-Garcia, X Ben, P Zhang, Z Lai, R Yan, X Zhai, W Meng, L Best-Rowden, A Jain, Q Cao, L Shen, W Xie, O Parkhi, A Zisserman, J Chen, Y Deng, G Bai, G Su, L Chen, J Zhao, J Deng, J Guo, N Xue, S Zafeiriou, T Dhamecha, G Goswami, R Singh, M Vatsa, M Ferrara, A Franco, D Maio, D Maltoni, Y Guo, L Zhang, Y Hu, X He, J Gao, A Hadid, M Pietikainen, K He, X Zhang, S Ren, J Sun, J Hernandez-Ortega, J Galbally, R Haraksim, L Beslay, E Learned-Miller, G Huang, A Roychowdhury, H Li, G Hua, Y Liu, Y Wang, R Yu, M Li, V Sharma, B Maze, J Adams, J Duncan, N Kalka, T Miller, C Otto, W Niggel, J Anderson, J Cheney, K Nasrollahi, T Moeslund, X Qi, C Liu, S Schuckers, U Saeed, J.-l Dugelay, N Belova, L Savchenko, F Schroff, D Kalenichenko, J Philbin, Y Sun, Y Chen, X Wang, X Tang, Y TaigmanOpenalex Unknown 2021 · DOI · OpenAlex

Abstract

The paper is considering the problem of increasing the performance and accuracy of video face identification. We examine the selection of the several best video frames using various techniques for assessing the quality of images. In contrast to traditional methods with estimation of image brightness/contrast, we propose to utilize the deep learning techniques that estimate the frame quality by using the lightweight convolutional neural network. In order to increase the effectiveness of the frame quality assessment step, we propose to distill knowledge of the cumbersome existing FaceQNet model for which there is no publicly available training dataset. The selected k-best frames are used to describe an input set of frames with a single average descriptor suitable for the nearest neighbor classifier. The proposed algorithm is compared with the traditional face feature extraction for each frame, as well as with the known clustering methods for a set of video frames.

Introduction

Video face recognition presents a significant computational challenge compared to static image recognition. A single video stream can contain thousands of frames; processing every frame through a deep neural network is slow and requires expensive hardware. Furthermore, more data isn't always better. In a typical video, many frames are blurry, poorly lit, or captured at awkward angles. Including these "junk" frames in the final calculation often confuses the system, leading to lower accuracy.

This research addresses the challenge of finding a way to pick the "best" frames (frame selection) using a method that is fast enough to run in real-time on standard hardware without sacrificing the nuance that deep learning provides. By distilling a heavy "teacher" model into a lightweight "student" CNN, researchers have developed a way to rapidly identify the highest-quality frames in a video stream.

Methodology

The authors propose a pipeline that prioritizes "quality over quantity." The core innovation lies in how they assess frame quality using knowledge distillation.

Knowledge Distillation

The gold standard for face quality assessment is a model called FaceQNet, but it’s based on a heavy ResNet-50 architecture—too slow for real-time video. The authors used FaceQNet as a "teacher" to train "student" models (based on MobileNet and LeNet) to predict quality scores. Since the original training data for FaceQNet isn't public, they used a subset of the VGGFace2 dataset and had the teacher "label" the images for the students.

The Recognition Pipeline

  1. Detection: Use MTCNN to find faces in the video.
  2. Quality Assessment: Run the distilled, lightweight CNN on every frame to get a quality score.
  3. Selection: Sort the frames and pick the top-$k$ best images (e.g., the best 1/8th of the total frames).
  4. Feature Extraction: Use a powerful model (like InsightFace or ResNet-50) to extract a mathematical "fingerprint" (descriptor) for only those $k$ frames.
  5. Aggregation: Instead of complex temporal modeling, they average the descriptors of the $k$ frames into one single vector and use a 1-Nearest Neighbor classifier to identify the person.
Comparison of quality scores ($Q_0, Q_1, Q_2$) for a clear frame (A) and an occluded frame (B). Higher scores indicate better quality for recognition.

Experimental Setup and Results

The researchers evaluated their approach on two major industry-standard benchmarks: IJB-C (IARPA Janus Benchmark-C) and YTF (YouTube Faces).

Key Insights

  • The "Noise" of Bad Frames: One of the most striking findings is that using all frames (the "Baseline") is actually worse than using a curated subset. Low-quality frames act as statistical noise that pulls the average descriptor away from the correct identity.
  • Distillation Efficiency: By using knowledge distillation, they created a "FaceQNet mobile" model that is roughly twice as fast as the original on a CPU while maintaining enough accuracy to pick the right frames.
  • The 1/8th Rule: For many datasets, selecting the top $N/8$ frames (where $N$ is the total number of frames) provided the best balance. It’s enough data to capture different angles of a face but strict enough to exclude the blur.

Performance Metrics

On the IJB-C dataset using a ResNet-50 extractor, the baseline accuracy (using all frames) was 70.89%. By using their "FaceQNet mobile" to select the best $N/8$ frames, accuracy jumped to 79.27%—a nearly 9% improvement. Because the quality assessment is so lightweight, the total time spent on a video is drastically reduced. Instead of running a heavy feature extractor 1,000 times for a 1,000-frame video, they run it only 125 times.

Conclusion

While effective, the paper notes a few areas for improvement. Current limitations include the use of a fixed $k$ value, the assumption of a single person per video, and the lack of temporal ordering in the aggregation step. Future work could explore adaptive selection methods or lightweight temporal models like GRUs to further refine accuracy and efficiency.

Summary updated

Abstract

The paper is considering the problem of increasing the performance and accuracy of video face identification.We examine the selection of the several best video frames using various techniques for assessing the quality of images.In contrast to traditional methods with estimation of image brightness/contrast, we propose to utilize the deep learning techniques that estimate the frame quality by using the lightweight convolutional neural network.In order to increase the effectiveness of the frame quality assessment step, we propose to distill knowledge of the cumbersome existing FaceQNet model for which there is no publicly available training dataset.The selected k-best frames are used to describe an input set of frames with a single average descriptor suitable for the nearest neighbor classifier.The proposed algorithm is compared with the traditional face feature extraction for each frame, as well as with the known clustering methods for a set of video frames.
BibTeX
@inproceedings{kharchevnikova2021peer,
  author = {Angelina Kharchevnikova and A. Savchenko and F Alonso-Fernandez and J Fierrez and J Ortega-Garcia and X Ben and P Zhang and Z Lai and R Yan and X Zhai and W Meng and L Best-Rowden and A Jain and Q Cao and L Shen and W Xie and O Parkhi and A Zisserman and J Chen and Y Deng and G Bai and G Su and L Chen and J Zhao and J Deng and J Guo and N Xue and S Zafeiriou and T Dhamecha and G Goswami and R Singh and M Vatsa and M Ferrara and A Franco and D Maio and D Maltoni and Y Guo and L Zhang and Y Hu and X He and J Gao and A Hadid and M Pietikainen and K He and X Zhang and S Ren and J Sun and J Hernandez-Ortega and J Galbally and J Fierrez and R Haraksim and L Beslay and A Kharchevnikova and A Savchenko and A Kharchevnikova and A Savchenko and E Learned-Miller and G Huang and A Roychowdhury and H Li and G Hua and Y Liu and Y Wang and R Yu and M Li and V Sharma and Y Wang and B Maze and J Adams and J Duncan and N Kalka and T Miller and C Otto and A Jain and W Niggel and J Anderson and J Cheney and K Nasrollahi and T Moeslund and X Qi and C Liu and S Schuckers and X Qi and C Liu and S Schuckers and U Saeed and J.-L Dugelay and A Savchenko and A Savchenko and A Savchenko and N Belova and L Savchenko and F Schroff and D Kalenichenko and J Philbin and Y Sun and Y Chen and X Wang and X Tang and Y Taigman and Andrey Savchenko},
  title = {{Peer Review \#1 of "Efficient video face recognition based on frame selection and quality assessment (v0.1)"}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.7287/peerj-cs.391v0.1/reviews/1},
}
Figure