FaceCoresetNet: Differentiable Coresets for Face Set Recognition

Gil Shapira, Yosi KellerAAAI 2024 · DOI

TL;DR

This paper introduces FaceCoresetNet, a novel approach to face set recognition that uses differentiable coreset selection to efficiently and effectively represent individuals from a collection of images and videos. By learning to select a small, representative subset of "core" images that balance quality and diversity, FaceCoresetNet achieves state-of-the-art accuracy on challenging benchmarks while significantly reducing computational complexity.

What Problem Does This Paper Address?

Face Set Recognition (FSR) aims to identify individuals from a collection of images and video frames, which is more challenging than traditional face recognition that uses a single image. The core difficulty lies in aggregating information from these potentially large and unordered sets of faces. Existing methods often struggle to balance two key policies: emphasizing high-quality images while down-weighting low-quality ones, and promoting diversity within the set to avoid redundancy (e.g., from video bursts). Furthermore, many aggregation methods are computationally expensive, especially for large templates, and some are not permutation-invariant, which is crucial for unordered sets. This paper addresses the need for an efficient, permutation-invariant, and quality-and-diversity-aware method for face set representation.

Methods

The authors propose FaceCoresetNet, a system built around a novel differentiable coreset selection mechanism.

  • Core-Template Selection: Instead of processing all images in a template, FaceCoresetNet selects a small subset called a "Core-Template" (CT) of size K. This selection is guided by a learned metric that balances image quality and diversity.
  • Quality Metric: Image quality is approximated by the norm of the extracted face features.
  • Diversity Metric: Cosine distance between features is used to measure appearance differences.
  • Quality-Aware Distance: A novel distance metric dq(fi, fj; y) = dc(fi, fj) * ||fj||^y is introduced, where y is a learned parameter that controls the trade-off between quality and diversity.
  • Differentiable Farthest Point Sampling (FPS): To make the selection process end-to-end trainable, the non-differentiable Argmax operation in traditional FPS is replaced with sampling from the Gumbel-Softmax distribution. This allows gradients to flow back through the selection process.
  • Initialization: The selection starts with the highest quality feature (largest norm) to ensure permutation invariance.
  • Attention Mechanism: The selected Core-Template features are then enhanced using:
  • Multi-Head Self-Attention (MHA): To model relationships within the Core-Template.
  • Multi-Head Cross-Attention: To allow the Core-Template features (as queries) to extract additional information from the full template (as keys and values). This is crucial for leveraging information missed by the coreset.
  • Norm Encoding: Feature norms are encoded and added to the features to explicitly incorporate quality information into the attention mechanism.
  • Aggregation: The enriched Core-Template features are aggregated (summed and normalized) to produce the final template descriptor.
  • Loss Function: The model is trained using the AdaFace loss, a standard FR loss that adaptively computes class margins based on image quality.

Datasets

  • Training: The model is trained on WebFace4M, a large-scale dataset containing millions of facial images. To simulate the richer templates found in FSR benchmarks, they online augment the training data to create templates with simulated video clips and photometric noise.
  • Evaluation: The model is evaluated on standard FSR benchmarks: IJB-B and IJB-C. These datasets contain templates with a mix of still images and video frames, representing challenging "in-the-wild" scenarios.

Key Insights and Ideas

  • Differentiable Coreset Selection: The core innovation is making coreset selection differentiable. This allows the model to learn what constitutes a good coreset directly from data, rather than relying on fixed heuristics.
  • Quality-Diversity Trade-off: The learned parameter y in the quality-aware distance metric is a key insight, enabling the model to dynamically balance the importance of image quality versus diversity based on the training data and task.
  • Linear Complexity for Attention: By applying attention mechanisms only to the small Core-Template (size K) and using it as queries for cross-attention with the full template (size N), the authors achieve O(N) complexity for the attention stage, a significant improvement over the O(N^2) of standard self-attention on full templates.
  • Coreset as "Enhanced Queries": Using the Core-Template features as queries in cross-attention is an effective way to leverage the full template's information without the computational burden of processing all its features directly.

Key Results

  • State-of-the-Art Accuracy: FaceCoresetNet achieves the best TAR@FAR=1e-6 on both IJB-B (52.56%) and IJB-C (91.12%) datasets, outperforming previous SOTA methods.
  • Computational Efficiency: The method achieves linear time complexity (O(N)) with respect to template size, making it significantly more efficient than quadratic methods like CAFace, especially for large templates. This is visualized in Figure 6, showing a clear linear trend for FaceCoresetNet versus a quadratic one for CAFace.
  • Ablation Studies:
    • Enabling differential coreset selection alone boosted accuracy from 38.45% to 48.01%.
  • Adding MHA self-attention increased accuracy to 49.24%.
  • Incorporating cross-attention further improved it to 51.71%.
  • Finally, adding Norm encoding pushed the accuracy to the SOTA 52.56%.
  • Optimal Core-Template Size: Experiments showed that a core-template size of K=3 was optimal and consistently provided the best results.
  • Trained y Parameter: Training the y parameter for the quality-diversity trade-off yielded the best results (91.12% on IJB-C TAR@FAR=1e-6) compared to fixed values (Table 3).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the method is efficient, it still relies on a pre-trained backbone for feature extraction.
  • The simulated video clips for training might not perfectly capture the nuances of real-world video data.
  • Future Work:
    • The authors suggest that the core concept of differentiable coreset selection with a learned metric could be extended to other domains requiring efficient feature fusion from large, unordered sets.
  • Further exploration of different attention mechanisms or aggregation strategies could be beneficial.

Summary updated

BibTeX
@inproceedings{shapira2024facecoresetnet,
  author = {Gil Shapira and Yosi Keller},
  title = {{FaceCoresetNet: Differentiable Coresets for Face Set Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28276},
}
Figure