Temporal Selective Max Pooling Towards Practical Face Recognition

Xiang Xiang

TL;DR

This paper tackles the challenge of robustly identifying individuals in videos, even with varying head poses and computational constraints. They propose a novel "pose-selective max pooling" method that intelligently selects key frames to represent a video's identity, achieving comparable accuracy to existing methods while significantly reducing computational cost.

What Problem Does This Paper Address?

Recognizing individuals from video footage is a crucial task, but it's hampered by two major issues:

  1. Head Pose Variation: Faces in real-world videos rarely maintain a consistent orientation relative to the camera. Standard deep learning features, while powerful, often struggle to disentangle identity from these pose variations, leading to misidentifications. Simply averaging features across all frames can dilute the identity signal.
  2. Computational Expense: Processing every single frame in a video for feature extraction and comparison is computationally intensive, making real-time applications challenging.

The authors aim to develop a method that can accurately measure identity similarity between videos by effectively handling pose variations and reducing the computational burden.

Methods

The core of their approach lies in a two-stage process:

  1. Pose-Selective Key Frame Extraction:
    • Pose Estimation: For each frame in a video, they estimate the 3D head pose (yaw, pitch, roll) using facial landmarks.
  2. Pose Quantization (K-Means): They cluster these estimated 3D poses into K groups using a modified K-means algorithm. This algorithm not only minimizes within-cluster distances (like standard K-means) but also aims to maximize between-cluster distances, encouraging diverse pose representation.
  3. Key Frame Selection: From each pose cluster, they select the frame whose pose is closest to the cluster's centroid. This ensures that the selected frames capture a diverse range of poses present in the video while reducing the total number of frames to K.
  4. Face Alignment: The selected key frames are then "frontalized" by warping them to a canonical pose, further normalizing for pose variations.
  5. Max Correlation Pooling for Similarity Measurement:
    • Deep Feature Extraction: For each of the K aligned key frames, they extract deep features using a pre-trained CNN (specifically, the VGG-face descriptor).
  6. Max Correlation Similarity: Instead of averaging these features, they measure the similarity between two videos by finding the maximum pairwise correlation between the feature vectors of their selected key frames. This "max pooling" approach is inspired by spatial pooling in CNNs, aiming to find the strongest evidence of similarity between any two frames from the respective videos.

Datasets

The primary dataset used for evaluation is the YouTube Faces (YTF) dataset. This dataset contains videos of individuals in uncontrolled environments, making it suitable for testing the robustness to pose variations. The authors note that they use an initial official list of 5,000 video pairs and later a corrected list of 4,999 pairs.

Key Insights and Ideas

  • Pose Diversity is Key: The paper argues that preserving the diversity of head poses within a video is more important than simply averaging features. This diversity acts as a proxy for the identity signal, especially when pose variations are significant.
  • K-Means for Pose Clustering: Using K-means to cluster poses is a clever way to group similar orientations and then select representative frames from each group. The modification to maximize between-cluster distance is crucial for capturing the full range of poses.
  • Max Correlation as a Similarity Metric: Moving beyond simple averaging, using the maximum correlation between frame features is a more robust way to find strong identity matches, especially when dealing with noisy or varying features. It's akin to finding the "best match" between any two frames from different videos.
  • Geometric Cues Enhance Deep Features: The paper highlights that incorporating geometric information (like pose) can significantly improve the descriptiveness of deep features, even without re-training the network.

Key Results

  • The proposed algorithm achieves comparable performance to VGG-face (which uses temporal mean pooling) on the YTF dataset for video-based face verification.
  • The Area Under the Curve (AUC) for the ROC curve on the YTF dataset is reported as 0.9419 (initial list) and 0.9418 (corrected list).
  • A significant advantage is the reduced computational expense due to selecting only K key frames instead of processing all frames.

Limitations and Future Work

  • Minor Pose Variations: The paper acknowledges that while their method handles significant pose variations, it might not be as effective for very subtle pose changes or when other factors like illumination are highly variable.
  • Face Expression Recognition: The authors suggest that their approach could be extended to other tasks like face expression recognition, where disentangling identity from expression would be the goal.
  • Further Datasets: They mention ongoing experiments on the BU-4DFE database, which contains acted facial expressions with moderate head pose variations, to further validate their approach.
  • Generalizability: The paper hints that the core idea of using geometric cues to improve deep feature descriptiveness could benefit other vision tasks beyond face recognition.

Summary updated

Abstract

In this paper, we deal with two challenges for measuring the similarity of the subject identities in practical video-based face recognition - the variation of the head pose in uncontrolled environments and the computational expense of processing videos. Since the frame-wise feature mean is unable to characterize the pose diversity among frames, we define and preserve the overall pose diversity and closeness in a video. Then, identity will be the only source of variation across videos since the pose varies even within a single video. Instead of simply using all the frames, we select those faces whose pose point is closest to the centroid of the K-means cluster containing that pose point. Then, we represent a video as a bag of frame-wise deep face features while the number of features has been reduced from hundreds to K. Since the video representation can well represent the identity, now we measure the subject similarity between two videos as the max correlation among all possible pairs in the two bags of features. On the official 5,000 video-pairs of the YouTube Face dataset for face verification, our algorithm achieves a comparable performance with VGG-face that averages over deep features of all frames. Other vision tasks can also benefit from the generic idea of employing geometric cues to improve the descriptiveness of deep features.
BibTeX
@inproceedings{xiang2016temporal,
  author = {Xiang Xiang},
  title = {{Temporal Selective Max Pooling Towards Practical Face Recognition}},
  booktitle = {VAAM/FFER@ICPR},
  year = {2016},
  url = {https://openreview.net/forum?id=wFUqXqPZ2d1b},
}
Figure