Pose-Selective Max Pooling for Measuring Similarity

Xiang Xiang, Trac D. Tran

TL;DR

This paper tackles the challenge of accurately measuring facial identity similarity in videos, especially when head poses vary significantly. They propose a novel "Pose-Selective Max Pooling" method that first selects a small set of "key frames" representing diverse head poses within a video, and then uses the maximum correlation between features from these key frames to determine identity similarity. This approach achieves comparable performance to existing methods while significantly reducing computational cost.

What Problem Does This Paper Address?

Measuring the similarity of facial identities across videos is crucial for applications like face verification and identification. A major hurdle in this task is the significant variation in head pose (the orientation of the face relative to the camera) that occurs naturally in uncontrolled video environments. Standard deep learning features, while powerful, are often not entirely robust to these pose variations, meaning a feature vector might capture both identity and pose information, making direct comparison unreliable. Simply averaging features across all frames in a video can dilute the identity signal and fail to capture the full range of pose variations. The paper aims to address this by disentangling pose variation from identity information to improve the accuracy and efficiency of video-based face recognition.

Methods

The core of the proposed method lies in two main stages:

  1. Pose-Selective Key Frame Selection:
    • Pose Estimation: For each frame in a video, facial landmarks are detected, and then used to estimate the 3D head pose (yaw, pitch, roll).
  2. Pose Quantization (K-means): The estimated 3D poses from all frames in a video are clustered using a modified K-means algorithm. This algorithm aims to minimize within-cluster distances (ensuring poses within a cluster are similar) while maximizing between-cluster distances (ensuring the selected clusters represent diverse poses).
  3. Key Frame Selection: From each pose cluster, the frame whose pose is closest to the cluster's centroid is selected. This results in a small set of K key frames that capture the overall pose diversity of the video.
  4. Face Alignment: The selected key frames are then "frontalized" by warping them to a canonical pose, further reducing pose-induced variations.
  5. Pose-Selective Max Pooling for Similarity Measurement:
    • Deep Feature Extraction: A pre-trained deep neural network (VGG-Face) is used to extract a high-dimensional feature vector from each of the K aligned key frames. This results in a "bag of features" for each video.
  6. Max Correlation Pooling: To measure the similarity between two videos (represented by their bags of features, $X_a$ and $X_b$), the paper computes the maximum pairwise correlation between all feature vectors from the two sets. This is formulated as: $Sim(X_a, X_b) = \max_{i,j} (f_{a,i}^T f_{b,j})$, where $f_{a,i}$ and $f_{b,j}$ are feature vectors from the key frames of video $a$ and $b$, respectively. This "max pooling" operation effectively finds the most similar pair of frames across the two videos, assuming that if two videos contain the same person, at least one pair of frames will exhibit high correlation.

Datasets

  • YouTube Faces (YTF) dataset: This is the primary dataset used for evaluation. It contains 3,425 videos of 1,595 subjects captured in uncontrolled environments. The authors specifically use an initial official list of 5,000 video pairs and a later corrected list of 4,999 video pairs for verification.

Key Insights and Ideas

  • Pose Diversity is Key: The paper highlights that preserving the diversity of head poses within a video, rather than simply averaging them, is crucial for robust identity representation.
  • K-means for Pose Clustering: Using K-means to cluster head poses provides an effective way to select a minimal set of representative frames that cover the pose variations.
  • Max Correlation as Similarity Metric: Employing the maximum pairwise correlation between frame features, rather than mean correlation, acts as a more robust measure of similarity, akin to finding nearest neighbors in feature space. This is particularly effective when dealing with potentially noisy or less discriminative frame pairs.
  • Computational Efficiency: By reducing hundreds of frames to a small set of K key frames, the computational cost of processing videos is significantly reduced, making real-time applications more feasible.

Key Results

The proposed "Pose-Selective Max Pooling" method achieves comparable performance to state-of-the-art methods like VGG-Face (which uses temporal mean pooling) on the YTF dataset. Specifically, the Area Under the Curve (AUC) for the Receiver Operating Characteristic (ROC) curve is reported as 0.9419 on the initial list of 5,000 video pairs and 0.9418 on a corrected list of 4,999 pairs. This indicates a high level of accuracy in distinguishing between same-person and different-person video pairs. The method also offers a significant reduction in computational complexity due to the key frame selection.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that not all landmarks can be perfectly aligned, and the frontalization process prioritizes salient landmarks. Profiles, for instance, are not frontalized.
  • The assumption of group sparsity might not hold under imperfect alignment, as noted in the context of expression recognition.
  • Future Work:
    • The authors suggest extending their work to video-based face expression recognition, which also involves disentangling identity from other factors.
  • Further experiments are planned on the BU-4DFE database, which contains acted facial expressions with moderate head pose variations, to explore variable disentanglement in real-world data.
  • The core idea of using geometric cues to improve deep feature descriptiveness is seen as a generalizable concept applicable to other vision tasks.

Summary updated

Abstract

In this paper, we deal with two challenges for measuring the similarity of the subject identities in practical video-based face recognition - the variation of the head pose in uncontrolled environments and the computational expense of processing videos. Since the frame-wise feature mean is unable to characterize the pose diversity among frames, we define and preserve the overall pose diversity and closeness in a video. Then, identity will be the only source of variation across videos since the pose varies even within a single video. Instead of simply using all the frames, we select those faces whose pose point is closest to the centroid of the K-means cluster containing that pose point. Then, we represent a video as a bag of frame-wise deep face features while the number of features has been reduced from hundreds to K. Since the video representation can well represent the identity, now we measure the subject similarity between two videos as the max correlation among all possible pairs in the two bags of features. On the official 5,000 video-pairs of the YouTube Face dataset for face verification, our algorithm achieves a comparable performance with VGG-face that averages over deep features of all frames. Other vision tasks can also benefit from the generic idea of employing geometric cues to improve the descriptiveness of deep features.
BibTeX
@inproceedings{xiang2016pose,
  author = {Xiang Xiang and Trac D. Tran},
  title = {{Pose-Selective Max Pooling for Measuring Similarity}},
  booktitle = {VAAM/FFER@ICPR},
  year = {2016},
  url = {https://openreview.net/forum?id=wFUqXqPZ2d1b},
}
Figure