TL;DR
This paper introduces MUSE, a novel neural architecture search (NAS) framework for multimodal fake news detection. MUSE dynamically adapts its architecture to better fuse information from text and images, significantly improving detection accuracy, especially for news with missing modalities.
What Problem Does This Paper Address?
The proliferation of fake news on social media presents a significant challenge. Detecting fake news often requires analyzing multiple modalities, such as text and images, but existing methods struggle with two key issues:
- Rigid Architectures: Many models use fixed architectures, which can lead to suboptimal fusion of information from different modalities and hinder performance.
- Incomplete Modalities: Real-world fake news often lacks one or more modalities (e.g., an article with text but no image). Models trained on complete multimodal data often fail to generalize well to these partial-modality scenarios.
Methods
The core of MUSE is its application of Neural Architecture Search (NAS) to dynamically optimize the model's architecture for fake news detection. MUSE employs a "triple path" approach:
- Multimodal Feature Extraction: Standard pre-trained models (language models for text, ResNet for images) are used to extract initial features. These are then refined using a "Global-Local Guidance" mechanism to capture richer context. Crucially, for samples with missing modalities, the corresponding feature extraction is effectively bypassed.
- Multimodal Feature Fusion: This is where MUSE's novelty lies. It uses three paths:
- Two Dynamic Paths: These paths leverage DARTS (Differentiable Architecture Search) to dynamically search for optimal neural network structures. They are designed to be flexible, adapting to different types of fake news and improving feature fusion. One path focuses on a linear sequence of operations, while the other explores a more complex sequence path.
- One Static Path: This path remains constant during training and is designed to capture latent, auxiliary information. It acts as a stable component, providing a baseline for detection.
- Fake News Prediction: The outputs from the three paths are combined using a linear layer, and a binary cross-entropy loss function is used for training.
Datasets
The paper evaluates MUSE on two publicly available datasets:
- WEIBO: A Chinese dataset containing fake and real news from a microblog rumor dispelling system. It includes samples with missing text and images.
- PHEME: An English dataset that specifically focuses on news with missing images.
Additionally, the authors create a WEIBO_Partial dataset by artificially removing one modality from complete WEIBO samples to specifically test the model's performance on partial-modality data.
Key Insights and Ideas
- NAS for Dynamic Adaptation: Applying NAS to fake news detection is a powerful way to overcome the limitations of fixed architectures, allowing the model to tailor itself to the specific characteristics of the data.
- Triple Path Synergy: The combination of dynamic (flexible, adaptive) and static (stable, auxiliary) paths proves effective. The dynamic paths handle the complexity of multimodal fusion, while the static path provides a consistent signal.
- Handling Missing Modalities is Crucial: The results on the WEIBO_Partial dataset highlight that models must be robust to incomplete data. MUSE's ability to gracefully handle missing modalities is a significant advantage over methods that struggle with such scenarios.
- Simplicity Can Be Better: Ablation studies suggest that retaining fewer operators (after NAS) can sometimes lead to better performance, indicating that overly complex architectures aren't always beneficial and can introduce noise.
Key Results
- MUSE consistently outperforms several state-of-the-art single-modal and multimodal baselines on both the WEIBO and PHEME datasets, achieving higher accuracy and F1 scores.
- On the WEIBO_Partial dataset, as the proportion of partial-modality news increases, MUSE's performance advantage becomes more pronounced. It significantly outperforms MCAN (a strong multimodal model) and a simple baseline, demonstrating its superior adaptability.
- Ablation studies confirm the effectiveness of each of the three paths (linear, sequence, auxiliary) in MUSE. Removing any path leads to a degradation in performance.
Limitations and Future Work
- Computational Cost of NAS: While MUSE leverages NAS for dynamic architecture search, the search process itself can be computationally intensive.
- Generalization to Other Modalities: The current MUSE model focuses on text and image modalities. Extending it to incorporate other modalities like video or audio would be a natural next step.
- Resource-Constrained Scenarios: The authors express interest in investigating the application of MUSE in resource-constrained environments, suggesting future work on model compression or efficiency.
Summary updated