TL;DR
This paper introduces Multimodal Channel-Mixing (MCM), a novel self-supervised pre-training framework for facial Action Unit (AU) detection. MCM leverages early fusion of RGB, depth, and thermal data with a masked autoencoder approach, specifically masking channels rather than spatial patches, to learn robust multimodal representations.
What Problem Does This Paper Address?
Facial Action Unit (AU) detection, crucial for understanding human expressions and emotions, traditionally relies on labeled data, which is expensive and time-consuming to acquire. While multimodal data (like RGB, depth, and thermal) offers richer information, effectively fusing these heterogeneous sources remains a challenge. Existing methods often use late or middle fusion, increasing model complexity and ignoring low-level inter-modal interactions. Early fusion, while simpler, has been underexplored, especially for learning channel-wise redundancies and handling missing modalities. This paper aims to bridge this gap by proposing a unified, efficient, and robust multimodal pre-training framework that can also adapt to missing modalities.
Methods
The core of the proposed method is the Multimodal Channel-Mixing (MCM) module. This module takes concatenated RGB, Depth, and Thermal (5 channels total) as input. It then randomly drops two of these channels and shuffles the remaining three, creating a "mixed" input. This mixed input is then fed into a Channel and Spatial Masked AutoEncoder.
The autoencoder consists of:
- ViT Encoder: A Vision Transformer (ViT) encoder processes the visible, unmasked patches of the mixed input. Unlike standard ViTs, it uses 2D sine-cosine positional embeddings.
- Spatial Masking: Similar to Masked Autoencoders (MAE), a large subset of patches is randomly masked out before encoding.
- ViT Decoders: Three separate ViT decoders are used to reconstruct the original RGB, Depth, and Thermal modalities from the encoded features and mask tokens. The decoders are designed to be asymmetric, with fewer layers than the encoder, inspired by MAE's efficiency.
The pre-training objective is to reconstruct the missing channels, forcing the model to learn inter-modal relationships and reduce channel redundancy. After pre-training, the encoder is fine-tuned for the downstream AU detection task.
Datasets
- BP4D+: Used for pre-training the MCM model.
- BP4D: Used for fine-tuning and evaluating the AU detection performance.
- DISFA: Used for fine-tuning and evaluating the AU detection performance.
Key Insights and Ideas
- Channel-Mixing for Multimodal Fusion: Instead of spatial masking, the paper proposes masking channels in a multimodal setting. This is a novel approach to exploit and learn from the inherent redundancy and complementarity across different modalities.
- Early Fusion with Channel Redundancy Reduction: The Channel-Mixing module effectively reduces channel redundancy while forcing the model to learn how to reconstruct missing information, leading to more robust representations.
- Unified Framework for Missing Modalities: The MCM framework is designed to be flexible. The pre-trained encoder can be fine-tuned on single modalities, demonstrating its ability to adapt even when some data sources are unavailable during fine-tuning.
- Efficiency through Single Encoder: By using a single encoder for all modalities during pre-training, MCM significantly reduces the number of trainable parameters compared to traditional late-fusion methods that require separate encoders for each modality.
Key Results
- State-of-the-Art Performance: The proposed MCM method achieves state-of-the-art results on both BP4D and DISFA datasets for AU detection, outperforming existing single-modal and multimodal methods.
- Single-Modal Superiority: On the BP4D dataset, MCM fine-tuned on RGB alone achieves the best performance among single-modal methods, outperforming several established baselines.
- Multimodal Advantage: When using fused RGB+Depth modalities, MCM achieves the highest F1-score (66.2%) on BP4D, outperforming other multimodal fusion methods.
- Effective Reconstruction: The reconstruction results shown in Figure 4 demonstrate that the model can effectively reconstruct original modalities from masked inputs, even with high masking ratios (up to 90%).
- Parameter Efficiency: MCM reduces trainable parameters by approximately 57% compared to traditional three-encoder late fusion methods.
Limitations and Future Work
- Spatial and Temporal Consistency: The paper acknowledges that the current approach requires modalities to be consistent in space and time. Future work will investigate feature-level channel mixing to address this.
- Color Bias in DISFA: The reconstruction quality on DISFA was noted to be lower than on BP4D due to color biases in the DISFA dataset, which the model trained on BP4D+ struggled to match. This highlights a potential challenge when applying models trained on one dataset to another with different data characteristics.
Summary updated