Pose-Selective Max Pooling for Measuring Similarity

Xiang Xiang, Trac D. Tran

Abstract

In this paper, we deal with two challenges for measuring the similarity of the subject identities in practical video-based face recognition - the variation of the head pose in uncontrolled environments and the computational expense of processing videos. Since the frame-wise feature mean is unable to characterize the pose diversity among frames, we define and preserve the overall pose diversity and closeness in a video. Then, identity will be the only source of variation across videos since the pose varies even within a single video. Instead of simply using all the frames, we select those faces whose pose point is closest to the centroid of the K-means cluster containing that pose point. Then, we represent a video as a bag of frame-wise deep face features while the number of features has been reduced from hundreds to K. Since the video representation can well represent the identity, now we measure the subject similarity between two videos as the max correlation among all possible pairs in the two bags of features. On the official 5,000 video-pairs of the YouTube Face dataset for face verification, our algorithm achieves a comparable performance with VGG-face that averages over deep features of all frames. Other vision tasks can also benefit from the generic idea of employing geometric cues to improve the descriptiveness of deep features.

Introduction: The Challenge of Video-Based Face Recognition

Recognizing faces in "wild" video environments, such as those found on YouTube, presents significant hurdles compared to static image recognition. Two major issues dominate:

  1. Pose Variation: Deep Convolutional Neural Networks (CNNs) are not perfectly pose-invariant. Averaging features across a video where a subject turns their head can "blur" identity information with pose noise.
  2. Computational Efficiency: Processing every frame in a long video is resource-intensive.
Pose variation in uncontrolled video environments often complicates identity verification.

The core insight of this research is that while pose varies throughout a video, identity remains constant. By intelligently selecting frames that represent the full spectrum of poses, we can decouple identity from motion.

Methodology

The proposed pipeline consists of two main stages: Pose Selection and Max Correlation Pooling.

Diversity-Preserving K-Means for Frame Selection

Rather than random sampling, the algorithm utilizes 3D geometric cues. For every frame, 3D rotation angles (yaw, pitch, and roll) are estimated. A modified K-means algorithm then partitions these poses into $K$ clusters, aiming to minimize intra-cluster distance (closeness) while maximizing inter-cluster distance (diversity). The frame closest to each cluster center is selected, reducing hundreds of frames to a representative "bag" of just $K$ features (the authors found $K=9$ to be optimal).

A representative frame selected to capture a specific pose orientation within the video sequence.

Max Correlation Pooling

To compare two videos, the authors employ Max Correlation instead of standard mean pooling. Every feature in Video A's bag is compared against every feature in Video B's bag. The maximum similarity score found across all pairs is used as the final metric. This approach assumes that if two videos depict the same person, there will be at least one pair of frames with similar poses where the identity match is strongest.

Experimental Results

The method was evaluated on the YouTube Face (YTF) dataset, which contains 3,425 videos of 1,595 individuals. On the official 5,000 video-pair verification task, the algorithm achieved an Area Under the Curve (AUC) of 0.9419. This performance is comparable to the VGG-Face model using temporal mean pooling across all frames, but with a fraction of the computational cost. The authors also noted that the method is robust to the "noise" of pose, as identity becomes the primary source of variation across compared videos.

Conclusion

This work demonstrates that integrating "old-school" geometric cues like 3D pose can significantly enhance the descriptiveness and efficiency of "new-school" deep learning features. By focusing on a small, diverse subset of frames, we can achieve high-accuracy face recognition suitable for real-time applications. Future work may extend this "selective pooling" concept to other facial attributes, such as expressions, or other vision tasks like action recognition.

Summary updated

Abstract

In this paper, we deal with two challenges for measuring the similarity of the subject identities in practical video-based face recognition - the variation of the head pose in uncontrolled environments and the computational expense of processing videos. Since the frame-wise feature mean is unable to characterize the pose diversity among frames, we define and preserve the overall pose diversity and closeness in a video. Then, identity will be the only source of variation across videos since the pose varies even within a single video. Instead of simply using all the frames, we select those faces whose pose point is closest to the centroid of the K-means cluster containing that pose point. Then, we represent a video as a bag of frame-wise deep face features while the number of features has been reduced from hundreds to K. Since the video representation can well represent the identity, now we measure the subject similarity between two videos as the max correlation among all possible pairs in the two bags of features. On the official 5,000 video-pairs of the YouTube Face dataset for face verification, our algorithm achieves a comparable performance with VGG-face that averages over deep features of all frames. Other vision tasks can also benefit from the generic idea of employing geometric cues to improve the descriptiveness of deep features.
BibTeX
@inproceedings{xiang2016pose,
  author = {Xiang Xiang and Trac D. Tran},
  title = {{Pose-Selective Max Pooling for Measuring Similarity}},
  booktitle = {VAAM/FFER@ICPR},
  year = {2016},
  url = {https://openreview.net/forum?id=wFUqXqPZ2d1b},
}
Figure