SHIELD: an evaluation benchmark for face spoofing and forgery detection with multimodal large language models

Introduction

Facial recognition systems face increasing threats from physical Face Anti-Spoofing (FAS) attacks, such as 3D masks and printed photos, and digital Face Forgery, including sophisticated Deepfakes. While traditional "black-box" models are often brittle when encountering novel attack vectors, Multimodal Large Language Models (MLLMs) offer a promising alternative due to their cross-modal reasoning capabilities. This paper introduces the SHIELD benchmark to evaluate the efficacy of MLLMs in detecting these subtle security threats.

The SHIELD Benchmark and Datasets

The SHIELD benchmark provides a rigorous testing environment for MLLMs, utilizing True/False and Multiple-Choice formats. It aggregates data from diverse sources to cover a wide range of attack modalities:

• Face Anti-Spoofing (FAS): Data from WMCA and SiW-Mv2 datasets, encompassing 352 and 348 identities respectively, with attacks ranging from screen replays to complex 3D masks.
• Forgery Detection: Data from FaceForensics++ (FF++) and the DeepFakeFace (DFF) dataset, including both GAN-based and Diffusion-based digital manipulations.
• Multimodal Inputs: Beyond standard RGB, the benchmark incorporates Infrared (IR) and Depth maps, testing the models' ability to synthesize non-visual data for security verification.

Expand

Source, p. 5

Methodology: MA-COT and Fine-Tuning

To enhance MLLM performance, the researchers propose two primary strategies:

1. Multi-Attribute Chain of Thought (MA-COT): Unlike standard reasoning, MA-COT directs the model to analyze specific, task-relevant attributes—such as pixel consistency, gloss, reflection, and texture naturalness—before reaching a verdict. This structured approach improves interpretability and significantly reduces "refusal" rates caused by safety filters in advanced models like GPT-4V.
2. Supervised Fine-Tuning (SFT): The study demonstrates that task-specific fine-tuning can transform general-purpose MLLMs into specialized detectors.

Experimental Results and Insights

Evaluation across various models reveals that MLLMs generally excel at identifying physical spoofing (e.g., paper masks) where semantic clues like lighting and edges are prominent. However, they struggle with the mathematical artifacts of digital forgeries.

Expand

Source, p. 3

Key findings include:

• Baseline Performance: Without specialized training, most MLLMs perform near chance on complex tasks. GPT-4V and Gemini emerged as leaders, with GPT-4V achieving 33.1% accuracy on FAS multiple-choice questions.
• Impact of SFT: Fine-tuning the LLaVA model resulted in a dramatic accuracy increase from 65.54% to 98.06% on FAS tasks, with a corresponding 26% reduction in the Half Total Error Rate (HTER).
• MA-COT Efficacy: For GPT-4V, the MA-COT paradigm boosted zero-shot FAS accuracy from 58.11% to 83.67% for specific attack types, highlighting the importance of structured prompting.
• Contextual Sensitivity: While few-shot prompting is often beneficial, it can lead to "cognitive overload" in smaller MLLMs, occasionally reducing accuracy due to information complexity.

Conclusion and Future Directions

While MLLMs show significant potential, challenges remain regarding computational costs and the need for high-quality, specialized pre-training data. Future work will focus on expanding datasets, exploring cross-task collaboration with other anomaly detection fields, and developing more sophisticated NLP metrics to evaluate the qualitative reasoning provided by these models.