MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing

Ajian Liu, Yanyan Liang

Abstract

Existing multi-modal face anti-spoofing (FAS) frameworks are typically designed using halfway or late fusion strategies. Halfway fusion requires test modalities to be consistent with training inputs, limiting deployment, while late fusion relies on multiple branches, increasing memory and computational requirements. We present the Modality-Agnostic Vision Transformer (MA-ViT), a single-branch framework designed to improve FAS performance across arbitrary modal attacks. MA-ViT utilizes early fusion to aggregate training modalities, enabling flexible testing of any given modal samples. Central to this architecture is the Modality-Agnostic Transformer Block (MATB), which employs Modal-Disentangle Attention (MDA) to eliminate modality-specific noise and Cross-Modal Attention (CMA) to supplement liveness features. Experiments show that MA-ViT outperforms existing single-modal frameworks and approaches multi-modal state-of-the-art performance with significantly lower FLOPs and parameters.

What Problem Does This Paper Address?

Face Anti-Spoofing (FAS) is critical for securing facial recognition systems against presentation attacks like photos, replays, or masks. While multi-modal data (RGB, Depth, IR) enhances accuracy, current fusion strategies face significant hurdles:

  1. Rigidity (Halfway Fusion): These models require the exact same sensors at test time as used during training. A missing sensor renders the model unusable.
  2. Computational Inflexibility (Late Fusion): These models use separate branches for each modality, leading to high memory usage and slow execution, which is unsuitable for edge devices.
Comparison of fusion strategies: (a) Halfway Fusion, (b) Late Fusion, and (c) Early Fusion (MA-ViT).

MA-ViT addresses these by creating a "modality-agnostic" system: a single-branch model that learns from multi-modal data but can be deployed flexibly regardless of the available sensors at runtime.

Methods

MA-ViT is built on a Vision Transformer (ViT) backbone, enhanced with components to handle multi-modal inputs within a single stream:

  • Early Fusion via Tokenization: All modalities are converted into tokens and fed into a single transformer branch. A learnable Modality Token (MOD) is introduced alongside the standard Class Token (CLS) to help the model distinguish the source sensor for each sequence.
  • Modality-Agnostic Transformer Block (MATB): This core component consists of two specialized attention mechanisms:
  • Modal-Disentangle Attention (MDA): Uses the MOD token to identify and mask features unique to a specific modality (e.g., color in RGB), forcing the model to focus on universal "liveness" features.
  • Cross-Modal Attention (CMA): Allows the model to supplement liveness features from one modality with cues from another (e.g., using depth information to verify RGB data).
  • Shared Classification Heads: Shared heads ensure that the final decision logic remains consistent, whether the input is single-modal or multi-modal.
The MA-ViT architecture overview, detailing the tokenization module, transformer encoder with MATB blocks, and classification heads.

Key Insights and Ideas

  • Subtraction is Addition: By explicitly identifying and removing modality-specific information via MDA, the model becomes more effective at isolating robust liveness features.
  • Single-Branch Efficiency: MA-ViT avoids the parameter explosion of multi-branch CNNs. It adds only ~1M parameters and 0.35G FLOPs over a standard ViT-Small while delivering superior performance.
  • Flexible Deployment: The model can be trained on high-end multi-sensor data but remains highly effective on devices equipped with only a standard RGB camera.

Key Results

MA-ViT consistently outperformed both single-modal baselines and complex multi-modal state-of-the-art (SOTA) methods across several benchmarks:

  • WMCA Dataset: Achieved an ACER of 3.81%, nearly halving the error of the previous best (7.60% by CMFL).
  • MmFA Dataset: Reduced the ACER from 6.51% (standard ViT) to 0.80%.
  • Generalization: In cross-testing scenarios, MA-ViT demonstrated superior robustness, suggesting that its learned liveness features are truly universal and not overfitted to specific sensor noise.
  • Efficiency: The model maintains a small footprint, making it a viable candidate for real-world mobile and edge applications.

Limitations and Future Work

  • Hyperparameter Sensitivity: The MDA module relies on a threshold (λ) to mask modality tokens. While λ=0.8 was optimal in this study, different sensor types may require further tuning.
  • Backbone Exploration: While tested on ViT-Small, Tiny, and Base, future work could explore more recent architectures like Swin Transformers or MobileViTs for further optimization.
  • Modality Scaling: Future research could investigate how the model scales when more than three sensor types (RGB, Depth, IR) are introduced simultaneously.

Summary updated

Abstract

The existing multi-modal face anti-spoofing (FAS) frameworks are designed based on two strategies: halfway and late fusion. However, the former requires test modalities consistent with the training input, which seriously limits its deployment scenarios. And the latter is built on multiple branches to process different modalities independently, which limits their use in applications with low memory or fast execution requirements. In this work, we present a single branch based Transformer framework, namely Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities’ data and enables flexible testing of any given modal samples. Further, we develop the Modality-Agnostic Transformer Block (MATB) in MA-ViT, which consists of two stacked attentions named Modal-Disentangle Attention (MDA) and Cross-Modal Attention (CMA), to eliminate modality-related information for each modal sequences and supplement modality-agnostic liveness features from another modal sequences, respectively. Experiments demonstrate that the single model trained based on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.
BibTeX
@inproceedings{liu2022ma,
  author = {Ajian Liu and Yanyan Liang},
  title = {{MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=rRPNzwoa4m},
}
Figure