Self-Supervised Distilled Learning for Multi-modal Misinformation Identification

Michael Mu, Sreyasee Das Bhattacharjee, Junsong YuanWACV 2023 · DOI

Abstract

Detecting "Cheapfakes"—real images paired with misleading captions—remains a significant challenge in misinformation research due to the lack of digital manipulation in the source media. We present SSDL (Self-Supervised Distilled Learner), a framework that leverages a two-phase learning process combining multi-task self-supervision with knowledge distillation. By aligning visual and textual modalities through task-agnostic and task-specific objectives, SSDL achieves a 7% improvement in identifying falsified samples over standard CLIP baselines. Furthermore, the model incorporates visual explainability, providing human-interpretable justifications for its classifications.

Introduction

While AI-generated "Deepfakes" often dominate public discourse, "Cheapfakes" or Out-of-Context (OOC) misinformation represent a more pervasive threat. In OOC scenarios, pristine images are repurposed with false textual narratives. Because these images contain no pixel-level artifacts or "glitches," traditional forensic methods fail. Current automated systems are often limited by the scarcity of massive, high-quality annotated datasets and the "black box" nature of deep learning models, which hinders verification by human fact-checkers.

Methodology

The SSDL framework addresses these challenges through a dual-phase training strategy designed to maximize learning from limited multimodal data.

Phase 1: Multi-task Pretraining

The model, built upon a CLIP backbone, undergoes simultaneous training on two fronts:

  1. Task-Agnostic Evaluation: Uses contrastive learning (SimCLRV2) to establish visual-visual and visual-text consistency ($L_{vis}^{self}$ and $L_{text}^{self}$), ensuring the model understands fundamental cross-modal relationships.
  2. Task-Specific Evaluation: Introduces a classification objective ($L_{comp}^{cls}$) to begin distinguishing between pristine and falsified samples.

Phase 2: Knowledge Distillation

To refine performance, a larger "Teacher" network ($f_{te}$) is pretrained and frozen. A smaller "Student" network ($f_{st}$) is then fine-tuned to mimic the Teacher’s probability distribution queue ($p_i^{te}$) through a knowledge distillation loss ($L_{dis}^{self}$). This allows the Student to capture complex, domain-specific nuances that might be missed during standard training.

The SSDL framework architecture, illustrating the two-phase learning process involving self-supervised pretraining and teacher-student knowledge distillation.

Experimental Results

Performance on NewsCLIPpings

SSDL was evaluated on the NewsCLIPpings Dataset, which features challenging splits based on semantic similarity, person matching, and scene context. On the comprehensive Balanced split, SSDL achieved a ~3% improvement in total accuracy compared to the CLIP baseline. Most notably, the model showed a 7% gain specifically in identifying "Falsified" samples, demonstrating superior sensitivity to subtle contextual mismatches.

Data Efficiency and Explainability

SSDL exhibits remarkable data efficiency, matching baseline performance while using only 50-75% of the training data. Beyond raw metrics, the framework utilizes LIME (Local Interpretable Model-Agnostic Explanations) to generate visual heatmaps. These segmentations highlight specific image regions—such as background objects or people—that contributed to a "Falsified" verdict, aiding human verification.

Qualitative results showing images, captions, ground truth labels, model predictions, and the corresponding explanation segmentations generated by the SSDL framework.

Conclusion

SSDL provides a robust, explainable approach to detecting out-of-context misinformation. By combining self-supervised alignment with knowledge distillation, the framework overcomes data scarcity and provides critical visual evidence for its decisions. Future research will explore extending this architecture to video-text content and integrating external world knowledge to detect even more subtle geographic or temporal inconsistencies.

Summary updated

BibTeX
@inproceedings{mu2023self,
  author = {Michael Mu and Sreyasee Das Bhattacharjee and Junsong Yuan},
  title = {{Self-Supervised Distilled Learning for Multi-modal Misinformation Identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00284},
}
Figure