TL;DR
This paper tackles the challenging problem of matching faces captured by different camera types (visible light vs. infrared) by proposing a novel multimodal ranking optimization approach. They introduce a network to translate images between modalities and then fuse multiple ranking lists to improve identification accuracy, outperforming existing methods on the SCface dataset.
What Problem Does This Paper Address?
The paper addresses the critical issue of heterogeneous face re-identification (Re-ID). This is the task of identifying the same person across different cameras that capture images in different spectral bands, specifically visible light (VIS) and near-infrared (NIR). While face Re-ID is important for surveillance, the significant "modality gap" between VIS and NIR images (they look very different) drastically degrades the performance of traditional Re-ID methods. Existing approaches often struggle to bridge this gap, leading to inaccurate matches.
Methods
The authors propose a two-stage method:
- Heterogeneous Face Image Translation: To reduce the modality gap, they employ a bidirectional generative network. This network learns to translate images from one modality to another (NIR to VIS, and VIS to NIR). Crucially, they introduce a "similarity pushing and dissimilarity pulling" training strategy. This involves using both positive (paired) and negative (unpaired) samples to train the network.
- Similarity Pushing: Encourages generated images to be more similar to their true counterparts.
- Dissimilarity Pulling: Encourages generated images to be dissimilar to images from other identities.
- This strategy aims to create more robust and complementary translated images, mitigating biases introduced by single-modality translation.
- Multimodal Ranking Optimization: After generating translated images, they create multiple sets of face pairs (e.g., original VIS-NIR, translated NIR-VIS, translated VIS-NIR). For each pair, they perform a bidirectional ranking (forward and backward ranking). This results in multiple initial ranking lists, each potentially capturing different aspects of similarity due to the varied image pairs.
- Linear Fusion: This algorithm combines the initial ranking lists by assigning weights. The weights are determined by the number of "k-reciprocal neighbors" in each list, assuming lists with more such neighbors are more reliable.
- Non-linear Fusion: This algorithm focuses on finding a representative rank by identifying a minimal "k" that makes images k-reciprocal neighbors across multiple ranking lists.
- Finally, a full fusion function combines the results from both linear and non-linear fusion to produce the final optimized re-ranked list.
Datasets
- SCface dataset: This is the primary dataset used for training and evaluation. The authors highlight that SCface is suitable because it contains low-resolution images with pose variations, mimicking real-world surveillance scenarios, unlike some other datasets that might have higher resolution or less variation. They split the dataset into training (65 pairs) and testing (30 pairs) subsets.
Key Insights and Ideas
- Modality Gap as an Opportunity: Instead of just seeing the VIS-NIR modality gap as a problem, the authors leverage it. By translating images and creating multiple ranking lists from different pairings, they exploit the complementarity of these biased lists to achieve better overall results.
- "Similarity Pushing and Dissimilarity Pulling" is Key: The novel training strategy for the translation network is crucial. It goes beyond simple image-to-image translation by actively pushing for similarity with the correct identity and pulling away from incorrect ones, leading to more discriminative translated features.
- Fusion of Biased Ranks: The core idea of fusing multiple, imperfect ranking lists (each biased in its own way) to produce a more accurate final ranking is a powerful concept. It suggests that combining diverse, even flawed, information can lead to a superior outcome.
- Bidirectional Ranking is Essential: Using both forward and backward ranking for each image pair provides richer information for the fusion process, capturing relationships from both query-to-gallery and gallery-to-query perspectives.
Key Results
- The proposed multimodal ranking optimization method significantly outperforms baseline methods and other related re-ranking algorithms on the SCface dataset.
- Specifically, the authors report improvements in CMC@1, CMC@5, and CMC@10 accuracies. For instance, their method achieves 50.63% Rank1, 66.26% Rank5, and 75.84% Rank10 on average across two subsets, which is substantially higher than the baseline (38.96% Rank1).
- Ablation studies show that both similarity pushing and dissimilarity pulling strategies contribute to performance gains, with their combination yielding the best results.
- The fusion of both linear and non-linear algorithms further boosts performance, especially for Rank5 and Rank10, compared to using either fusion method alone.
Limitations and Future Work
- Dataset Limitations: The authors acknowledge that existing NIR-VIS face databases, including SCface, are often collected for face recognition and may not fully capture real-world surveillance challenges like significant pose variations and occlusions.
- Future Work: They suggest collecting a more reliable face image database specifically designed for heterogeneous face Re-ID that better accounts for these real-world factors.
Summary updated