Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning

Zuhui Wang, Zhaozheng Yin, Young Anna Argyris

Abstract

Vaccine hesitancy, fueled by misinformation on social media, remains a significant public health challenge. While text-based detection has been studied extensively on platforms like Twitter, image-centric platforms like Instagram present unique challenges where misinformation is often embedded in visual content, OCR text, and specific hashtags. This paper introduces a multimodal deep learning framework that analyzes these components simultaneously. By leveraging a novel "semantic- and task-level" attention mechanism (SeTa_Att) and an ensemble of models, the proposed system achieves a 97.4% accuracy rate in identifying antivaccine content, significantly outperforming traditional unimodal and state-of-the-art multimodal methods.

The Challenge of Image-Centric Misinformation

As misinformation migrates from text-heavy platforms to visual ones like Instagram, traditional automated moderation tools often fall short. On Instagram, the "antivaccine" message is rarely confined to the caption alone. Instead, it is frequently embedded within the image itself as overlaid text or hidden within a dense string of community-specific hashtags.

Figure 1: Examples of Instagram post components including images, OCR-extracted text, captions, and hashtags across different post types.

Existing systems that analyze text or images in isolation fail to capture this full context. Users often intentionally overlay text on images to subvert automated text filters. To address this, a robust detection system must "see" and "read" a post holistically, integrating visual features with both explicit captions and implicit metadata like hashtags.

Methodology: The Three-Branch Multimodal Network

The researchers developed a three-branch multimodal deep learning architecture designed to process the distinct data streams of an Instagram post.

Figure 3: The proposed three-branch multimodal architecture featuring Hashtag, Text, and Image branches integrated via the SeTa_Att mechanism.

Architectural Components

  1. Hashtag Branch: This branch utilizes fine-tuned fastText embeddings. This approach is specifically chosen to handle the "out-of-vocabulary" nature of hashtags (e.g., #vaccineinjury), which often concatenate multiple words into unique tokens.
  2. Text Branch: This branch processes a combination of the post's caption and text extracted from the image via Optical Character Recognition (OCR). This combined corpus is analyzed using a bidirectional RNN with Gated Recurrent Units (GRU).
  3. Image Branch: Visual features are extracted using a fine-tuned VGG19 network. This is further enhanced by a "spatial and channel squeeze-and-excitation" (scSE) block, which helps the model focus on the most semantically relevant regions of the image.

Semantic- and Task-level Attention (SeTa_Att)

The core innovation of this framework is the SeTa_Att mechanism. While standard attention mechanisms focus on semantic meaning, SeTa adds a "task" layer. This layer weighs features based on their similarity to known antivaccine markers, prioritizing hashtags or visual cues that historically correlate with misinformation rather than general sentiment.

Ensemble Classification

To maximize performance, the authors employ an ensemble method. Prediction scores from the three-branch model are combined with scores from three separate single-modal models. These are fed into a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel to produce the final classification.

Dataset and Experimental Setup

The framework was trained and validated on a real-world dataset of 31,282 Instagram posts collected between 2016 and 2019. The dataset was balanced (50% antivaccine, 50% non-antivaccine) and curated using a "snowballing" technique that identified influential accounts starting from core keywords like #bigpharma. Ground truth was established through independent labeling by three trained annotators using a majority-voting scheme.

Key Insights and Results

The experimental results demonstrate that multimodality is essential for high-accuracy detection in visual social media environments:

  • Superior Accuracy: The ensemble model achieved a benchmark accuracy of 97.4% (0.978 Precision, 0.967 Recall), outperforming existing multimodal models like MVAE (92.9%) and EANN (84.8%).
  • The Power of OCR: Integrating OCR into the text branch proved vital for neutralizing subversion tactics where users hide text within images to dodge filters.
  • Ablation Studies: While the multimodal approach was superior, studies showed the caption-only model (91.1% accuracy) still outperformed the image-only model (86.5%), indicating that textual signals remain the most explicit indicators of intent.
  • Real-World Efficacy: In a 30-day live test conducted in late 2019, the model successfully tracked the daily volume of antivaccine posts on Instagram with high precision, proving its viability for active monitoring.

Limitations and Future Directions

While highly effective, the model faces challenges with posts requiring deep external context (e.g., specific legislative bills like "SB 792") and highly stylized or handwritten OCR text. Future work may involve incorporating "memory cells" to track evolving misinformation tactics over time and improving the system's ability to interpret localized political or social contexts.

Summary updated

BibTeX
@article{wang2021detecting,
  author = {Zuhui Wang and Zhaozheng Yin and Young Anna Argyris},
  title = {{Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2021},
  url = {https://openreview.net/forum?id=61jXsipjOC},
}
Figure