TL;DR
This paper introduces WMMT, a novel framework for pinpointing precisely when and which modality (visual, audio, or both) is forged in Deepfake videos, using only video-level labels. By employing multitask learning and a unique "deviation perceiving" loss, WMMT achieves performance comparable to fully supervised methods, even with limited annotations.
What Problem Does This Paper Address?
The proliferation of Deepfake videos poses a significant threat to trust and social stability. While detecting Deepfakes is challenging, accurately localizing when a forgery occurs within a video, especially when only one modality (like visual or audio) is manipulated, is even harder. Existing methods often require detailed frame-level annotations, which are costly and time-consuming to obtain. Furthermore, the field has seen limited research on weakly supervised approaches that can perform this fine-grained temporal localization using only coarse, video-level labels, particularly when dealing with multimodal forgeries (manipulations affecting both visual and audio streams). This paper tackles this gap by proposing a method for Weakly Supervised Multimodal Temporal Forgery Localization (WS-MTFL).
Methods
The core of the proposed WMMT framework is Multitask Learning (MTL). Instead of training separate models for visual and audio forgery detection, WMMT treats them as related tasks within a single model. This allows the model to learn shared representations and leverage commonalities between modalities, improving overall performance.
Key novel components include:
- Feature Enhancement Module with Temporal Property Preserving Attention (TPPA): This module aims to capture subtle forgery traces by focusing on both intra-modality (within visual or within audio) and inter-modality (between visual and audio) feature deviations. It uses an attention mechanism to enhance features while preserving their temporal integrity.
- Extensible Deviation Perceiving Loss: This loss function is designed to exploit the inherent temporal characteristics of Deepfakes. The authors observe that forged segments often exhibit greater temporal deviation (e.g., sudden changes or inconsistencies) between adjacent frames compared to genuine content. This loss encourages the model to amplify these deviations in forged samples and suppress them in genuine ones, providing a powerful supervisory signal for weakly supervised learning.
- Modality-Aware Expert Selection: During inference, a "Mixture-of-Experts" (MoE) inspired mechanism adaptively selects the most relevant features and localization head based on the model's initial prediction of the forgery type (visual only, audio only, or both). This helps to reduce interference between modalities and improve localization accuracy.
Datasets
The authors evaluate their method on two challenging datasets:
- LAV-DF: A content-driven multimodal forgery dataset with a significant number of genuine and forged videos.
- AV-Deepfake1M: A large-scale multimodal forgery dataset featuring a diverse range of subjects and forgery types.
Both datasets provide video-level annotations, aligning with the paper's weakly supervised approach.
Key Insights and Ideas
- Multitask Learning is Key for Multimodal Weak Supervision: The paper strongly advocates for MTL as a fundamental strategy for WS-MTFL. It effectively balances the need to model individual modality characteristics with the requirement to understand cross-modal interactions.
- Temporal Deviation is a Powerful Forgery Cue: The observation that forged segments exhibit greater temporal "jumpiness" than genuine content is a crucial insight. This deviation is effectively leveraged by the proposed loss function to guide the model without explicit frame-level labels.
- Adaptive Modality Selection Boosts Robustness: The MoE-like mechanism for selecting features and localization heads during inference is vital for handling the inherent asymmetry and complexity of multimodal forgeries. It allows the model to dynamically focus on the most informative modality.
- Weak Supervision Can Rival Full Supervision: The results demonstrate that WMMT, despite using only video-level labels, achieves performance comparable to fully supervised methods, highlighting the potential of well-designed weakly supervised techniques.
Key Results
- On the LAV-DF dataset, WMMT achieved an average mAP of 73.27% and an average AR of 85.51%, outperforming other weakly supervised methods and approaching fully supervised baselines.
- On the AV-Deepfake1M dataset, WMMT achieved an average mAP of 74.91% and an average AR of 85.51%, again demonstrating strong performance in a more challenging scenario.
- Ablation studies confirmed the significant contributions of each proposed component: MTL, feature enhancement, and the deviation perceiving loss, with MTL showing particularly substantial improvements.
- t-SNE visualizations showed that WMMT effectively enhances the separability of features for forged vs. genuine content across audio, visual, and multimodal representations.
Limitations and Future Work
- Deviation Measure Function: While the paper proposes an extensible deviation perceiving loss, the authors acknowledge that the choice of the specific deviation measure function (e.g., MSE, L1, cosine) can influence performance. Investigating more effective functions is suggested as a future research direction.
- Generalizability: While cross-dataset evaluation showed promising generalization, a noticeable performance drop occurred. Further work could focus on improving model robustness across diverse datasets and forgery generation techniques.
- Computational Complexity: The paper mentions the use of pre-trained models for feature extraction, and while the proposed framework is efficient, further optimization for real-time applications might be explored.
- Exploration of More Strategies: The authors suggest exploring more strategies to improve model generalizability, indicating that while WMMT is effective, there's still room for advancement in this complex domain.
Summary updated