Abstract
Existing multi-modal face anti-spoofing (FAS) frameworks are typically designed using halfway or late fusion strategies. Halfway fusion requires test modalities to be consistent with training inputs, limiting deployment, while late fusion relies on multiple branches, increasing memory and computational requirements. We present the Modality-Agnostic Vision Transformer (MA-ViT), a single-branch framework designed to improve FAS performance across arbitrary modal attacks. MA-ViT utilizes early fusion to aggregate training modalities, enabling flexible testing of any given modal samples. Central to this architecture is the Modality-Agnostic Transformer Block (MATB), which employs Modal-Disentangle Attention (MDA) to eliminate modality-specific noise and Cross-Modal Attention (CMA) to supplement liveness features. Experiments show that MA-ViT outperforms existing single-modal frameworks and approaches multi-modal state-of-the-art performance with significantly lower FLOPs and parameters.
What Problem Does This Paper Address?
Face Anti-Spoofing (FAS) is critical for securing facial recognition systems against presentation attacks like photos, replays, or masks. While multi-modal data (RGB, Depth, IR) enhances accuracy, current fusion strategies face significant hurdles:
- Rigidity (Halfway Fusion): These models require the exact same sensors at test time as used during training. A missing sensor renders the model unusable.
- Computational Inflexibility (Late Fusion): These models use separate branches for each modality, leading to high memory usage and slow execution, which is unsuitable for edge devices.
MA-ViT addresses these by creating a "modality-agnostic" system: a single-branch model that learns from multi-modal data but can be deployed flexibly regardless of the available sensors at runtime.
Methods
MA-ViT is built on a Vision Transformer (ViT) backbone, enhanced with components to handle multi-modal inputs within a single stream:
- Early Fusion via Tokenization: All modalities are converted into tokens and fed into a single transformer branch. A learnable Modality Token (MOD) is introduced alongside the standard Class Token (CLS) to help the model distinguish the source sensor for each sequence.
- Modality-Agnostic Transformer Block (MATB): This core component consists of two specialized attention mechanisms:
- Modal-Disentangle Attention (MDA): Uses the MOD token to identify and mask features unique to a specific modality (e.g., color in RGB), forcing the model to focus on universal "liveness" features.
- Cross-Modal Attention (CMA): Allows the model to supplement liveness features from one modality with cues from another (e.g., using depth information to verify RGB data).
- Shared Classification Heads: Shared heads ensure that the final decision logic remains consistent, whether the input is single-modal or multi-modal.
Key Insights and Ideas
- Subtraction is Addition: By explicitly identifying and removing modality-specific information via MDA, the model becomes more effective at isolating robust liveness features.
- Single-Branch Efficiency: MA-ViT avoids the parameter explosion of multi-branch CNNs. It adds only ~1M parameters and 0.35G FLOPs over a standard ViT-Small while delivering superior performance.
- Flexible Deployment: The model can be trained on high-end multi-sensor data but remains highly effective on devices equipped with only a standard RGB camera.
Key Results
MA-ViT consistently outperformed both single-modal baselines and complex multi-modal state-of-the-art (SOTA) methods across several benchmarks:
- WMCA Dataset: Achieved an ACER of 3.81%, nearly halving the error of the previous best (7.60% by CMFL).
- MmFA Dataset: Reduced the ACER from 6.51% (standard ViT) to 0.80%.
- Generalization: In cross-testing scenarios, MA-ViT demonstrated superior robustness, suggesting that its learned liveness features are truly universal and not overfitted to specific sensor noise.
- Efficiency: The model maintains a small footprint, making it a viable candidate for real-world mobile and edge applications.
Limitations and Future Work
- Hyperparameter Sensitivity: The MDA module relies on a threshold (λ) to mask modality tokens. While λ=0.8 was optimal in this study, different sensor types may require further tuning.
- Backbone Exploration: While tested on ViT-Small, Tiny, and Base, future work could explore more recent architectures like Swin Transformers or MobileViTs for further optimization.
- Modality Scaling: Future research could investigate how the model scales when more than three sensor types (RGB, Depth, IR) are introduced simultaneously.
Summary updated