Attacking Face Recognition With T-Shirts: Database, Vulnerability Assessment, and Detection

Mathias Ibsen, Christian Rathgeb, Fabian Brechtel, Ruben Klepp, Katrin Pöppelmann, Anjith George, Sébastien Marcel, Christoph Busch

This paper, "Attacking Face Recognition with T-shirts: Database, Vulnerability Assessment and Detection," explores a novel and low-cost method for spoofing face recognition systems: printing target faces on T-shirts. The researchers introduce a new dataset (TFPA), demonstrate that current recognition systems are highly vulnerable to this "hiding in plain sight" attack, and propose new multi-modal detection strategies to counter it.

TL;DR

Researchers have identified that T-shirts with faces printed on them are a highly effective and easily concealed way to fool face recognition systems, achieving over a 92% success rate against state-of-the-art models. To combat this, they released the first dedicated T-shirt attack database (TFPA) and proposed a fusion-based detection method that significantly outperforms existing anti-spoofing techniques.

What Problem Does This Paper Address?

Face recognition systems are increasingly used for security, but they are susceptible to Presentation Attacks (PAs)—attempts to subvert the system using tools like printed photos, digital screens, or 3D masks. While much research has focused on rigid masks or high-resolution replays, the "T-shirt attack" has been largely overlooked.

T-shirts represent a unique threat because they are non-rigid, inexpensive to produce, and can be easily concealed (e.g., under a cardigan) until the moment of the attack. Furthermore, because the fabric drapes over a human body, it creates subtle depth variations that can trick simple 3D sensors that might otherwise easily catch a flat paper print. The authors address the lack of data and standardized evaluation for this specific attack vector.

Methods

The authors approached the problem in three stages:

  1. Attack Generation: They used StyleGAN to generate 100 high-quality synthetic identities. These were printed on T-shirts worn by eight different subjects.
  2. Vulnerability Assessment: They tested how well standard face detectors (like MTCNN and RetinaFace) and recognizers (like ArcFace and a commercial-off-the-shelf system) handled these images across eight different scenarios (e.g., looking left/right, stretching the shirt, or wearing a facial mask to hide the attacker's real face).
  3. Detection (PAD): They proposed three new Presentation Attack Detection (PAD) methods:
  4. Depth Variance (DV): A non-learning method that analyzes the statistical variance of depth values at specific facial landmarks.
  5. Anomaly Detection (AD): A model trained only on "bona fide" (real) RGB images using features from a Vision Transformer (ViT) and a one-class SVM.
  6. Fusion: A score-level combination of the DV and AD approaches.

Datasets

  • TFPA (T-shirt Face Presentation Attack) Database: The core contribution. It contains 1,608 attack images using 100 unique T-shirt prints, captured in both visible (RGB) and depth spectra using an Intel RealSense D435 camera.
  • HQ-WMCA: Used to train baseline state-of-the-art PAD models for comparison.
  • FRGCv2: Used to provide bona fide (real) face data for training the anomaly detection model.

Key Insights and Ideas

  • The "Success" of Face Detectors is a Weakness: Standard face detectors are too good. They easily locate the face on the T-shirt with high confidence, passing it along to the recognition engine as a valid input.
  • Generalization is the Primary Hurdle: PAD systems trained on traditional attacks (like silicon masks or paper prints) fail to recognize T-shirt attacks as fraudulent. This highlights a "generalization gap" where security systems are blind to attack types they haven't seen during training.
  • Depth is the Deciding Factor: While a T-shirt face looks "real" in 2D, its depth profile is significantly flatter and more uniform than a human face. By looking at the variance of depth across landmarks, the system can distinguish between a 3D head and a 2D print on a non-rigid surface.

Key Results

  • High Vulnerability: Both ArcFace and the commercial system (COTS) were highly vulnerable, with an Impostor Attack Presentation Match Rate (IAPMR) exceeding 92.6% when the attacker's real face was concealed.
  • Baseline Failure: Existing SOTA detection methods (like CMFL) trained on other datasets struggled, showing an error rate (D-EER) of about 24-26% on T-shirt attacks.
  • Proposed Solution: The authors' proposed Fusion (DV + AD) approach achieved a much better D-EER of 12.52%, demonstrating that combining depth statistics with RGB anomaly detection is a robust defense strategy.

Limitations and Future Work

The authors acknowledge that their study focused on white T-shirts in controlled indoor environments. Future work could explore different fabric colors, textures, and more complex "in-the-wild" lighting conditions.

Additionally, they noted that if a face recognition system were to analyze the area outside the immediate face (such as the neck and shoulders), T-shirt attacks might be easier to spot. However, most current systems crop tightly to the face for efficiency, which inadvertently aids the attacker. Exploring "context-aware" detection that looks at the transition between the face and the body is a promising future direction.

Summary updated

Abstract

Face recognition systems are widely deployed for biometric authentication. Despite this, it is well-known that, without any safeguards, face recognition systems are highly vulnerable to presentation attacks. In response to this security issue, several promising methods for detecting presentation attacks have been proposed which show high performance on existing benchmarks. However, an ongoing challenge is the generalization of presentation attack detection methods to unseen and new attack types. To this end, we propose a new T-shirt Face Presentation Attack (TFPA) database of 1,608 T-shirt attacks using 100 unique presentation attack instruments. In an extensive evaluation, we show that this type of attack can compromise the security of face recognition systems and that some state-of-the-art attack detection mechanisms trained on popular benchmarks fail to robustly generalize to the new attacks. Further, we propose three new methods for detecting T-shirt attack images, one which relies on the statistical differences between depth maps of bona fide images and T-shirt attacks, an anomaly detection approach trained on features only extracted from bona fide RGB images, and a fusion approach which achieves competitive detection performance.
BibTeX
@inproceedings{ibsen2023attacking,
  author = {Mathias Ibsen and Christian Rathgeb and Fabian Brechtel and Ruben Klepp and Katrin Pöppelmann and Anjith George and Sébastien Marcel and Christoph Busch},
  title = {{Attacking Face Recognition With T-Shirts: Database, Vulnerability Assessment, and Detection}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=7Amxp9ZXta},
}

Loading related content...

Figure