Abstract
The paper is considering the problem of increasing the performance and accuracy of video face identification. We examine the selection of the several best video frames using various techniques for assessing the quality of images. In contrast to traditional methods with estimation of image brightness/contrast, we propose to utilize the deep learning techniques that estimate the frame quality by using the lightweight convolutional neural network. In order to increase the effectiveness of the frame quality assessment step, we propose to distill knowledge of the cumbersome existing FaceQNet model for which there is no publicly available training dataset. The selected k-best frames are used to describe an input set of frames with a single average descriptor suitable for the nearest neighbor classifier. The proposed algorithm is compared with the traditional face feature extraction for each frame, as well as with the known clustering methods for a set of video frames.
Introduction
Video face recognition presents a significant computational challenge compared to static image recognition. A single video stream can contain thousands of frames; processing every frame through a deep neural network is slow and requires expensive hardware. Furthermore, more data isn't always better. In a typical video, many frames are blurry, poorly lit, or captured at awkward angles. Including these "junk" frames in the final calculation often confuses the system, leading to lower accuracy.
This research addresses the challenge of finding a way to pick the "best" frames (frame selection) using a method that is fast enough to run in real-time on standard hardware without sacrificing the nuance that deep learning provides. By distilling a heavy "teacher" model into a lightweight "student" CNN, researchers have developed a way to rapidly identify the highest-quality frames in a video stream.
Methodology
The authors propose a pipeline that prioritizes "quality over quantity." The core innovation lies in how they assess frame quality using knowledge distillation.
Knowledge Distillation
The gold standard for face quality assessment is a model called FaceQNet, but it’s based on a heavy ResNet-50 architecture—too slow for real-time video. The authors used FaceQNet as a "teacher" to train "student" models (based on MobileNet and LeNet) to predict quality scores. Since the original training data for FaceQNet isn't public, they used a subset of the VGGFace2 dataset and had the teacher "label" the images for the students.
The Recognition Pipeline
- Detection: Use MTCNN to find faces in the video.
- Quality Assessment: Run the distilled, lightweight CNN on every frame to get a quality score.
- Selection: Sort the frames and pick the top-$k$ best images (e.g., the best 1/8th of the total frames).
- Feature Extraction: Use a powerful model (like InsightFace or ResNet-50) to extract a mathematical "fingerprint" (descriptor) for only those $k$ frames.
- Aggregation: Instead of complex temporal modeling, they average the descriptors of the $k$ frames into one single vector and use a 1-Nearest Neighbor classifier to identify the person.
Experimental Setup and Results
The researchers evaluated their approach on two major industry-standard benchmarks: IJB-C (IARPA Janus Benchmark-C) and YTF (YouTube Faces).
Key Insights
- The "Noise" of Bad Frames: One of the most striking findings is that using all frames (the "Baseline") is actually worse than using a curated subset. Low-quality frames act as statistical noise that pulls the average descriptor away from the correct identity.
- Distillation Efficiency: By using knowledge distillation, they created a "FaceQNet mobile" model that is roughly twice as fast as the original on a CPU while maintaining enough accuracy to pick the right frames.
- The 1/8th Rule: For many datasets, selecting the top $N/8$ frames (where $N$ is the total number of frames) provided the best balance. It’s enough data to capture different angles of a face but strict enough to exclude the blur.
Performance Metrics
On the IJB-C dataset using a ResNet-50 extractor, the baseline accuracy (using all frames) was 70.89%. By using their "FaceQNet mobile" to select the best $N/8$ frames, accuracy jumped to 79.27%—a nearly 9% improvement. Because the quality assessment is so lightweight, the total time spent on a video is drastically reduced. Instead of running a heavy feature extractor 1,000 times for a 1,000-frame video, they run it only 125 times.
Conclusion
While effective, the paper notes a few areas for improvement. Current limitations include the use of a fixed $k$ value, the assumption of a single person per video, and the lack of temporal ordering in the aggregation step. Future work could explore adaptive selection methods or lightweight temporal models like GRUs to further refine accuracy and efficiency.
Summary updated