Improved likelihood ratios for face recognition in surveillance video by multimodal feature pairing

Andrea Macarulla Rodríguez, Zeno Geradts, Marcel Worring, Luis UnzuetaForensic 2024 · DOI

TL;DR

This paper tackles the challenge of reliable face recognition in surveillance videos for forensic applications by proposing a multimodal feature pairing approach. By integrating face image quality (FIQ) and attribute information, they significantly improve the estimation of likelihood ratios (LRs), a crucial metric for evidence strength, outperforming previous methods and even human experts in some cases.

What Problem Does This Paper Address?

In forensic investigations, accurately identifying individuals from surveillance footage is paramount. However, traditional manual comparisons are subjective and lack standardization, creating a critical gap in evidence reliability. While deep learning has advanced still-image face recognition, video-based recognition, especially in challenging real-world scenarios (varying pose, illumination, occlusions), remains a significant hurdle. Existing automated systems often struggle to quantify the strength of evidence, which is where Likelihood Ratios (LRs) come in. LRs provide a probabilistic measure of how much the evidence supports one hypothesis (e.g., the person in the video is the suspect) over another (e.g., it's a different person). The challenge lies in accurately estimating these LRs from video data, especially when dealing with low-quality frames and the need for a standardized, evaluative forensic framework.

Methods

The core of this research is a novel "multimodal feature pairing" approach for image-to-video face recognition. Instead of just comparing raw face embeddings, they enrich the process by incorporating:

  • Face Image Quality (FIQ): They use quality assessment models (SER-FIQ and SDD-FIQA) to understand how good a frame is for recognition.
  • Facial Attributes: They extract various attributes like gender, pose (yaw, pitch, roll), ethnicity, expression, and presence of accessories (glasses, headgear).

The workflow involves:

  1. Multimodal Feature Pairing: This is the key innovation. They explore several pairing strategies:
  2. Highest Number of Common Attributes: Pairing frames that share the most attributes.
  3. Highest Quality Images: Prioritizing frames with better image quality.
  4. Weighted by Quality: Creating a combined embedding for the video and reference images, where each frame's contribution is weighted by its quality.
  5. Biometric Score Generation: Standard face recognition models (ArcFace, FaceNet, QMagFace) generate similarity scores (s) for these paired frames.
  6. Calibration: The raw scores are transformed into Likelihood Ratios (LRs) using calibration methods, specifically considering within-source variability (WSV) and between-source variability (BSV) from a calibration database.
  7. Validation: The estimated LRs are validated using the log-likelihood ratio cost (Cllr) metric, which measures both discrimination and calibration accuracy.

A significant experiment also involved preprocessing facial images with the super-resolution model CodeFormer to see its impact.

Datasets

The study utilizes a combination of established and specialized datasets:

  • ENFSI proficiency test [23]: Used for validation, involving mugshot images and CCTV video with 18 participants.
  • ForenFace [13]: Contains video sequences and extracted images of 97 subjects with detailed manual annotations.
  • SCFace [27]: Images from an uncontrolled indoor environment with five surveillance cameras.
  • XQLFW [21]: A variant of LFW focusing on cross-quality scenarios, emphasizing degraded images.
  • ChokePoint [22]: Real-world surveillance data with variations in illumination, pose, and sharpness.

Key Insights and Ideas

  • Quality is King (and Attributes Help): Prioritizing high-quality frames and pairing based on shared attributes significantly improves LR estimation, leading to more reliable forensic evidence.
  • Multimodal Fusion is Powerful: Combining face image quality with attribute information provides a richer representation than using either alone, leading to better discrimination.
  • Super-Resolution Can Backfire: Surprisingly, applying advanced super-resolution techniques like CodeFormer to enhance image quality decreased the reliability of LR estimation in this forensic context, suggesting it can introduce artifacts that confuse recognition systems.
  • Calibration Matters: The choice of calibration data (random vs. same-attribute vs. same-quality) can impact the final LR estimates, highlighting the need for careful selection.
  • Model Choice is Crucial: Different face recognition models (ArcFace, FaceNet, QMagFace) respond differently to various pairing and quality assessment strategies, emphasizing that there's no one-size-fits-all solution.

Key Results

  • The best-performing method, using the QMagFace model with SER-FIQ for quality assessment and pairing based on the highest number of common attributes, achieved a Cllr of 0.13. This is notably better than the average Cllr of 0.46 achieved by human participants in the ENFSI test.
  • A combined embedding weighted by frame quality was the second-best method, demonstrating the effectiveness of this approach.
  • Using highest-quality frames yielded similar Cllr values to the top 25% best frames approach.
  • The super-resolution preprocessing with CodeFormer unexpectedly increased the Cllr, indicating it undermined evidence reliability for this specific application.
  • The study found that pose (yaw) significantly impacts image quality, with profile poses generally resulting in lower quality. Interestingly, gender did not show a significant influence on image quality.

Limitations and Future Work

  • Attribute Estimation Errors: The paper acknowledges that errors in attribute estimation in real-world surveillance can affect methods relying on shared attributes. Future work should investigate how these errors impact performance and explore alternative pairing strategies that are less sensitive to them.
  • Generalizability: While diverse datasets were used, further testing on even more varied datasets and with different face recognition algorithms is suggested to ensure generalizability.
  • Ethnic Bias: The authors highlight the importance of exploring the impact of ethnicity on face recognition systems, particularly concerning potential biases, and suggest this as a crucial area for future research.
  • Alternative Metrics: While Cllr was chosen for its comprehensive evaluation, exploring other metrics like Tippett or Empirical Cross-Entropy plots could offer deeper insights, though they might introduce complexity.
  • Super-Resolution Impact: The negative impact of CodeFormer warrants further investigation to understand why it degraded performance and if specific adjustments could mitigate this.

Summary updated

Abstract

In forensic and security scenarios, accurate facial recognition in surveillance videos, often challenged by variations in pose, illumination, and expression, is essential. Traditional manual comparison methods lack standardization, revealing a critical gap in evidence reliability. We propose an enhanced images-to-video recognition approach, pairing facial images with attributes like pose and quality. Utilizing datasets such as ENFSI 2015, SCFace, XQLFW, ChokePoint, and ForenFace, we assess evidence strength using calibration methods for likelihood ratio estimation. Three models-ArcFace, FaceNet, and QMagFace-undergo validation, with the log-likelihood ratio cost (<i>C</i><sub><i>llr</i></sub>) as a key metric. Results indicate that prioritizing high-quality frames and aligning attributes with reference images optimizes recognition, yielding similar <i>C</i><sub><i>llr</i></sub> values to the top 25% best frames approach. A combined embedding weighted by frame quality emerges as the second-best method. Upon preprocessing facial images with the super resolution CodeFormer, it unexpectedly increased <i>C</i><sub><i>llr</i></sub>, undermining evidence reliability, advising against its use in such forensic applications.
BibTeX
@inproceedings{rodriguez2024improved,
  author = {Andrea Macarulla Rodríguez and Zeno Geradts and Marcel Worring and Luis Unzueta},
  title = {{Improved likelihood ratios for face recognition in surveillance video by multimodal feature pairing}},
  booktitle = {Forensic},
  year = {2024},
  doi = {10.1016/j.fsisyn.2024.100458},
}
Figure