TL;DR
This paper introduces a novel approach to video-based face recognition by leveraging the Earth Mover's Distance (EMD) to compare distributions of facial features extracted from video sequences. By first reducing dimensionality with Fisher's Linear Discriminant Analysis (FLDA) and then representing each video as a "signature" of clustered features, the method achieves robust and efficient recognition, outperforming existing techniques.
What Problem Does This Paper Address?
Recognizing faces from video sequences presents unique challenges compared to still-image recognition. While multiple frames offer richer temporal information, issues like poor video quality, varying illumination, pose changes, partial occlusions, and small image sizes can significantly degrade performance. Existing methods often fall into two categories: sequential approaches that rely on temporal continuity and batch approaches that treat videos as independent sets of images. While batch approaches are useful for sparse data, traditional methods like Mutual Subspace Method (MSM) and K-L divergence can be computationally expensive and struggle with the complex, non-convex data distributions arising from dynamic facial variations. This paper aims to bridge this gap by proposing a more robust and efficient batch approach that effectively models these complex distributions.
Methods
The core of this paper's contribution lies in its novel application of Earth Mover's Distance (EMD) for video-based face recognition. The proposed method involves several key steps:
- Dimensionality Reduction: To combat the "curse of dimensionality" and improve efficiency, Fisher's Linear Discriminant Analysis (FLDA) is employed. FLDA transforms the high-dimensional image space into a lower-dimensional feature space, making the data more compact and separable by maximizing between-class variance while minimizing within-class variance.
- Signature Representation: Instead of directly modeling the complex data distribution of face images in the feature space, each video is represented as a "signature." This signature is created by clustering the feature vectors from the video using a K-Means algorithm. Each cluster is characterized by its mean feature vector and its weight (representing the proportion of features in that cluster). This signature effectively captures the overall distribution of features within a video.
- Earth Mover's Distance (EMD): The similarity between two videos is then computed using EMD. EMD is a distance metric that measures the minimum "work" required to transform one distribution (represented as a signature) into another. It's conceptually like moving earth from piles to fill holes, where the cost is proportional to the distance moved and the amount of earth. This metric is particularly well-suited for comparing distributions and can handle partial matching, which is beneficial when some faces in a video are of poor quality or irrelevant.
- Similarity Function: A final similarity score between 0 and 1 is derived from the EMD using an exponential function, allowing for easy interpretation.
Datasets
The authors used a combined database for their experiments, consisting of:
- Mobo (Motion of Body) database: Collected at Carnegie Mellon University, this database contains 25 individuals with 300 face images each.
- Collected database: This part was gathered from advertisements, MTV, and personal videos, also containing 25 subjects with 300 face images each.
In total, the combined database comprises 50 subjects, with face images normalized to 30x30 pixels after rough eye-position alignment. The database exhibits variations in poses and expressions.
Key Insights and Ideas
- EMD as a Distributional Metric: The paper highlights EMD's strength in comparing complex data distributions, which is crucial for handling the dynamic nature of video-based face recognition. Unlike simpler metrics, EMD can capture the "shape" of feature distributions.
- FLDA for Discriminative Features: FLDA proves to be an effective preprocessing step, not only reducing dimensionality but also enhancing the separability of features between different individuals, which is vital for accurate recognition.
- Signature Representation for Robustness: Representing videos as signatures (clusters of features with weights) offers a compact and flexible way to model data distributions, tolerating variations in pose, illumination, and expression.
- EMD's Advantage in Partial Matching: The ability of EMD to handle partial matching is a significant advantage, allowing the algorithm to effectively disregard less informative or corrupted facial frames within a video sequence.
- Signature vs. Gaussian Models: The paper suggests that signatures, due to their clustering-based approach, are more adept at capturing complex, non-convex data distributions compared to simpler models like Gaussian distributions used in FLDA+K-L.
Key Results
The experimental results demonstrate the effectiveness of the proposed FLDA+EMD approach:
- Superior Performance: FLDA+EMD consistently outperformed other methods, including Mutual Subspace Method (MSM), K-L divergence in original image space (K-L), and K-L divergence in FLDA feature space (FLDA+K-L), across various experimental settings.
- Feature Dimensionality Impact: The FLDA+EMD method showed the best recognition performance, especially when using fewer than 8 eigenvectors or features. While increasing features didn't significantly improve performance for FLDA+EMD, it did benefit FLDA+K-L.
- Set Size Impact: In experiments varying the size of training and testing sets, FLDA+EMD achieved the highest recognition rates, with rates reaching 100% in some configurations.
- FLDA's Effectiveness: The comparison between FLDA+K-L and K-L, and FLDA+EMD and EMD, clearly shows that FLDA is an effective method for dimensionality reduction and making features more discriminative.
Limitations and Future Work
- Signature Updating: The authors suggest that future work could explore updating methods to improve the representative capability of signatures.
- Temporal Information Integration: While this paper focuses on a batch approach, the authors acknowledge that incorporating temporal information and transformation probabilities (as seen in [4]) could lead to more robust video models.
- EMD as a Metric: The paper notes that EMD with partial matching is not strictly a metric. While it performs well for classification, this theoretical aspect might be a consideration for future theoretical development.
Summary updated