MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing

Ajian Liu, Yanyan Liang

Abstract

The existing multi-modal face anti-spoofing (FAS) frameworks are designed based on two strategies: halfway and late fusion. However, the former requires test modalities consistent with the training input, which seriously limits its deployment scenarios. And the latter is built on multiple branches to process different modalities independently, which limits their use in applications with low memory or fast execution requirements. In this work, we present a single branch based Transformer framework, namely Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities data and enables flexible testing of any given modal samples. Further, we develop the Modality-Agnostic Transformer Block (MATB) in MA-ViT, which consists of two stacked attentions named Modal-Disentangle Attention (MDA) and Cross-Modal Attention (CMA), to eliminate modality-related information for each modal sequences and supplement modality-agnostic liveness features from another modal sequences, respectively. Experiments demonstrate that the single model trained based on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.

Introduction: The Rigidity of Multi-Modal Fusion

Face Anti-Spoofing (FAS) is a critical security technology designed to prevent unauthorized access via presentation attacks, such as high-resolution photos, video replays, or sophisticated 3D masks. While multi-modal systems—utilizing RGB, Depth, and Infrared (IR) sensors—offer superior security, they often suffer from deployment rigidity.

Existing frameworks typically follow one of two strategies:

  1. Halfway Fusion: These models integrate features in the middle of the network. However, they require the exact same set of sensors at test time as were used during training. A model trained on RGB+Depth cannot function if only an RGB camera is available.
  2. Late Fusion: These models process each modality through independent branches, fusing them only at the final decision stage. While flexible, this multi-branch approach significantly increases memory usage and computational overhead, making it unsuitable for mobile or resource-constrained devices.
Comparison of multi-modal fusion strategies: (a) Halfway Fusion, (b) Late Fusion, and (c) the proposed Early Fusion approach.

MA-ViT addresses these issues by adopting an Early Fusion strategy within a single-branch Vision Transformer (ViT) architecture, allowing for flexible testing with any available modality while maintaining high efficiency.

Methods: Modality-Agnostic Architecture

The core innovation of MA-ViT is the Modality-Agnostic Transformer Block (MATB). Unlike standard ViT blocks, MATB is designed to extract "liveness" features that are universal across different sensors while filtering out modality-specific noise.

The MA-ViT architecture, featuring the Tokenization Module and the Modality-Agnostic Transformer Block (MATB) with MDA and CMA layers.

Key Components

  • Tokenization & Early Fusion: All input modalities are projected into a single joint embedding space at the beginning of the network. This allows a single-branch architecture to handle multi-modal data seamlessly.
  • Modal-Disentangle Attention (MDA): This layer utilizes a learnable "modality token" to identify and mask features specific to a particular sensor (e.g., the specific texture of a depth map). By disentangling these modality-related signals, the model is forced to focus on universal liveness cues.
  • Cross-Modal Attention (CMA): CMA enables interaction between different modalities. It allows the features of one modality to be supplemented by complementary information from others, ensuring that the most reliable spoofing traces are captured regardless of the input source.

Evaluation and Results

The researchers evaluated MA-ViT on three major multi-modal FAS benchmarks: CASIA-SURF (MmFA), CASIA-SURF CeFA, and WMCA.

Performance Highlights

  • State-of-the-Art Accuracy: On the WMCA dataset, MA-ViT achieved an Average Classification Error Rate (ACER) of 3.81%, a significant improvement over the previous best method, CMFL, which recorded 7.60%.
  • Flexible Testing Gains: A key finding was that training with multiple modalities improves performance even when testing on a single modality. When trained on RGB+Depth+IR but tested only on RGB, MA-ViT reduced the error rate by 9.57% compared to models trained solely on RGB data.
  • Efficiency: MA-ViT maintains a small footprint, adding only approximately 1M parameters and 0.35G FLOPs compared to a standard single-modality ViT, making it highly suitable for mobile deployment.

Conclusion

MA-ViT represents a significant step forward in making multi-modal face anti-spoofing practical for real-world applications. By moving to a modality-agnostic, single-branch architecture, it provides the flexibility to handle varying sensor availability without the computational burden of multi-branch networks. Future research will focus on handling noisy or low-quality sensor data and extending the framework to temporal video sequences to counter dynamic presentation attacks.

Summary updated

Abstract

The existing multi-modal face anti-spoofing (FAS) frameworks are designed based on two strategies: halfway and late fusion. However, the former requires test modalities consistent with the training input, which seriously limits its deployment scenarios. And the latter is built on multiple branches to process different modalities independently, which limits their use in applications with low memory or fast execution requirements. In this work, we present a single branch based Transformer framework, namely Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities data and enables flexible testing of any given modal samples. Further, we develop the Modality-Agnostic Transformer Block (MATB) in MA-ViT, which consists of two stacked attentions named Modal-Disentangle Attention (MDA) and Cross-Modal Attention (CMA), to eliminate modality-related information for each modal sequences and supplement modality-agnostic liveness features from another modal sequences, respectively. Experiments demonstrate that the single model trained based on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.
BibTeX
@inproceedings{liu2022ma,
  author = {Ajian Liu and Yanyan Liang},
  title = {{MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=rRPNzwoa4m},
}
Figure