Temporal Similarity Analysis of Remote Photoplethysmography for Fast 3D Mask Face Presentation Attack Detection

Siqi Liu, Xiangyuan Lan, Pong C. YuenWACV 2020 · OpenReview

TL;DR

This paper introduces a novel method, TSrPPG, for detecting 3D mask presentation attacks on face recognition systems using remote photoplethysmography (rPPG). By analyzing the temporal similarity of rPPG signals in the time domain, TSrPPG achieves high accuracy with significantly shorter observation times (as little as 1 second) compared to existing rPPG methods, making it practical for real-world applications like smartphone unlocks.

What Problem Does This Paper Address?

Face recognition systems are increasingly vulnerable to sophisticated presentation attacks, particularly those using 3D masks. While remote photoplethysmography (rPPG) – a technique that extracts heartbeat signals from subtle skin color changes captured by a regular camera – has shown promise as a liveness cue, existing rPPG-based methods require long video sequences (10-12 seconds) to reliably detect the heartbeat signal. This extended observation time is impractical for many applications, such as quick smartphone unlocks or e-payments, where user experience is paramount. The challenge, therefore, is to leverage rPPG for 3D mask detection with significantly reduced latency.

Methods

The core innovation of this paper is the Temporal rPPG Similarity (TSrPPG) feature operator. Unlike previous rPPG methods that rely on frequency domain analysis of long signals, TSrPPG operates in the time domain and focuses on the similarity of local rPPG signals. The key ideas are:

  • Local rPPG Signals: rPPG signals are extracted from multiple local facial regions identified by facial landmarks.
  • Temporal Similarity: The method analyzes how these local rPPG signals from different facial regions correlate with each other over short time intervals. For genuine faces, these signals originate from the same underlying heartbeat and thus exhibit temporal similarity in shape and phase. For masked faces, the signals are essentially random noise, leading to dissimilarities.
  • Feature Operator: A novel operator, TSrPPG, is proposed to quantify this temporal similarity. It involves calculating distances between shifted versions of local rPPG signals.
  • Facial vs. Background Comparison: To further enhance discriminability, TSrPPG also computes the similarity between local facial rPPG signals and signals extracted from background regions. Genuine facial rPPG should be less similar to background noise than the noisy signals from a masked face.
  • Multi-faceted Similarity Metrics: Three types of similarity metrics are employed:
  • Amplitude Similarity: Uses Euclidean distance to compare the amplitude variations of local rPPG signals.
  • Gradient Similarity: Employs normalized cross-correlation on the gradient of the rPPG signals to capture finer waveform details.
  • Phase Similarity: Uses the lag argument of the maximum cross-correlation to measure phase alignment.
  • Score-Level Fusion: The scores from these three similarity metrics are fused using a weighted sum to produce the final classification.

Datasets

The authors utilize and extend existing datasets to evaluate their method:

  • 3DMAD [10]: Used for evaluation.
  • HKBU-MARsV1+ [19]: Used for evaluation.
  • CSMAD [3]: Used to evaluate robustness to masks with different transmittance and challenging lighting.
  • HKBU-MARsV2+: A newly extended dataset. It builds upon the HKBU-MARsV2 dataset by adding 4 subjects and their customized REAL-f masks, expanding the total to 16 subjects and 10 high-quality masks. It also includes 6 lighting conditions and a total of 480 videos.

The paper emphasizes that their experiments are conducted on a large combined dataset of 11092 short-term video slots, covering significant variations in mask type, lighting, camera, resolution, and compression.

Key Insights and Ideas

  • Time-Domain rPPG for Speed: The most significant insight is that analyzing the temporal similarity of rPPG signals in the time domain, rather than relying on frequency spectrum analysis of longer signals, is key to achieving fast detection.
  • Heartbeat as a Universal Liveness Cue: The fundamental principle that genuine faces exhibit heartbeat-driven rPPG signals, while masks do not, remains the core liveness cue. The novelty lies in how this cue is extracted efficiently.
  • Local Signal Similarity is Discriminative: The subtle, yet consistent, temporal patterns in local rPPG signals from genuine faces are more discriminative than global spectral features when observation time is limited.
  • Background Noise as a Counter-Signal: Using background rPPG signals as a reference point for comparison with facial rPPG is an effective strategy to highlight the "artificiality" of masked face signals.
  • Multi-faceted Similarity is Robust: Combining amplitude, gradient, and phase similarity metrics provides a more comprehensive view of the rPPG signal, making the system robust to various real-world variations.

Key Results

  • Dramatic Speedup: The proposed TSrPPG method achieves high performance with an observation time of just 1 second, a significant improvement over the 10-12 seconds required by prior rPPG-based methods.
  • State-of-the-Art Performance: TSrPPG outperforms existing rPPG-based 3D mask PAD methods by a large margin, showing improvements of about 20% in EER and AUC on datasets like 3DMAD.
  • Generalizability: The method demonstrates strong generalizability across different datasets and challenging conditions, including variations in mask type, lighting, and camera settings. For instance, on HKBU-MARsV2+, TSrPPG achieves an AUC of 90.4% with 1-second observation, significantly outperforming baselines.
  • Robustness to Variations: The paper shows that TSrPPG is robust to variations in mask transmittance, eyeglass occlusion, and lighting conditions, as demonstrated on the CSMAD dataset.

Limitations and Future Work

  • Compression Artifacts: The paper notes that video compression (e.g., H.264) can significantly impact rPPG signal quality by removing subtle color variations, posing a challenge for rPPG-based PAD.
  • Need for More Diverse Datasets: The authors suggest that more self-made 3D mask attack datasets with diverse camera and compression settings are needed to further investigate the properties of rPPG-based PAD.
  • Future Directions: The paper frames TSrPPG as a general framework that can accommodate different signal similarity metrics and fusion strategies, suggesting potential for further refinement and adaptation. The robustness to compression artifacts is a clear area for future research.

Summary updated

Abstract

To tackle the 3D mask face presentation attack, remote Photoplethysmography (rPPG), a biomedical technique that can detect heartbeat signal remotely, is employed as an intrinsic liveness cue. Although existing rPPG-based methods exhibit encouraging results, they require long observation time (10-12 seconds) to identify the heartbeat information, which limits their employment in real applications such as smartphone unlock and e-payment. To shorten the observation time (within 1-second) while keeping the performance, we propose a fast rPPG-based 3D mask presentation attack detection (PAD) method by analyzing the similarity of local facial rPPG signals in the time domain. In particular, a set of temporal similarity features of facial and background local rPPG signals are designed and fused to adapt the real world variations based on rPPG shape and phase properties. For better evaluation under practical variations, we build the HKBU-MARsV2+ dataset that includes 16 masks from 2 types and 6 lighting conditions. Finally, extensive experiments are conducted on 11092 shortterm video slots from 4 datasets with a large number of real- world variations, in terms of mask type, lighting condition, camera, resolution of face region, and compression setting. Results show that the proposed TSrPPG outperforms the state-of-the-art competitors dramatically on discriminabil- ity and generalizability. To our best knowledge, this is the first work that addresses the length of observation time issue of rPPG-based 3D mask PAD.
BibTeX
@inproceedings{liu2020temporal,
  author = {Siqi Liu and Xiangyuan Lan and Pong C. Yuen},
  title = {{Temporal Similarity Analysis of Remote Photoplethysmography for Fast 3D Mask Face Presentation Attack Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=ESWxaRPVE1a},
}
Figure