Template-based Multi-Domain Face Recognition

Anirudh Nanduri, Rama Chellappa

Introduction

Standard face recognition systems perform exceptionally well under controlled, visible-light conditions. However, their accuracy degrades significantly in "non-visual" domains—such as thermal (LWIR), short-wave infrared (SWIR), or long-range surveillance—where images are frequently blurry, noisy, or of low resolution.

Face detection and alignment challenges at 300m, 400m, and 500m distances.

Most existing solutions rely on Domain Adaptation, which requires massive labeled datasets for every specific environmental condition. This paper addresses the "single-source, multi-target" problem: adapting a model trained solely on visible light to work across multiple challenging domains without extensive retraining.

Methodology: Beyond Average Pooling

The researchers focus on template-based recognition, where a subject is represented by a collection of images or video frames rather than a single static shot. The traditional approach, "Average Pooling," treats all frames equally, allowing a single low-quality or "junk" frame to pollute the entire template and reduce accuracy.

The template-based face recognition pipeline, from media set to probe template.

To solve this, the authors propose two novel algorithms that weight images based on their inherent quality:

  1. Feature Norm as Quality Proxy: Modern face recognition models (like ArcFace or AdaFace) naturally produce feature vectors whose $L_2$ norm correlates with image quality. Clearer faces result in "longer" vectors, while degraded images produce "shorter" ones.
  2. Norm Pooling (NP): This method applies a softmax function to feature norms to calculate weights. High-quality images contribute more to the final template, while low-quality ones are suppressed.
  3. Sparse Pooling (SP): Utilizing the sparsemax function, this variant can assign a weight of exactly zero to extremely poor frames. This allows the system to completely discard "junk" data that would otherwise degrade the template.

Experimental Results

The methods were evaluated using the IARPA JANUS Benchmark Multi-domain Face (IJB-MDF), a grueling dataset featuring visible and infrared imagery at distances up to 500 meters.

Key performance metrics (Rank-1 retrieval rates) include:

  • Visible 500m: Norm Pooling achieved 86.01%, a significant improvement over the 81.12% baseline of Average Pooling.
  • SWIR 30m: AdaFace combined with Norm Pooling reached 79.68%, outperforming standard averaging (77.29%).

Key Insights

  • Inherent Quality Signals: The feature extractor itself signals quality through vector magnitude; no separate quality assessment model is required.
  • Sparsity Benefits: Sparse Pooling proved highly effective, sometimes discarding over 70% of frames in a template to focus only on the highest-quality data.
  • Model Synergy: Models trained with quality-aware loss functions, such as AdaFace, showed the strongest performance gains when paired with Norm Pooling.

Conclusion and Future Work

While these pooling methods significantly bridge the gap in SWIR and long-range visible domains, challenges remain in thermal (MWIR/LWIR) spectra where the domain gap is too vast for visible-light models. Future research will focus on automatically estimating the scaling parameters ($\lambda$) used in these algorithms to further generalize across diverse environments.

Summary updated

BibTeX
@inproceedings{nanduri2024template,
  author = {Anirudh Nanduri and Rama Chellappa},
  title = {{Template-based Multi-Domain Face Recognition}},
  booktitle = {IJCB},
  year = {2024},
  doi = {10.1109/ijcb62174.2024.10744520},
}
Figure