Abstract
Detecting "Cheapfakes"—real images paired with misleading captions—remains a significant challenge in misinformation research due to the lack of digital manipulation in the source media. We present SSDL (Self-Supervised Distilled Learner), a framework that leverages a two-phase learning process combining multi-task self-supervision with knowledge distillation. By aligning visual and textual modalities through task-agnostic and task-specific objectives, SSDL achieves a 7% improvement in identifying falsified samples over standard CLIP baselines. Furthermore, the model incorporates visual explainability, providing human-interpretable justifications for its classifications.
Introduction
While AI-generated "Deepfakes" often dominate public discourse, "Cheapfakes" or Out-of-Context (OOC) misinformation represent a more pervasive threat. In OOC scenarios, pristine images are repurposed with false textual narratives. Because these images contain no pixel-level artifacts or "glitches," traditional forensic methods fail. Current automated systems are often limited by the scarcity of massive, high-quality annotated datasets and the "black box" nature of deep learning models, which hinders verification by human fact-checkers.
Methodology
The SSDL framework addresses these challenges through a dual-phase training strategy designed to maximize learning from limited multimodal data.
Phase 1: Multi-task Pretraining
The model, built upon a CLIP backbone, undergoes simultaneous training on two fronts:
- Task-Agnostic Evaluation: Uses contrastive learning (SimCLRV2) to establish visual-visual and visual-text consistency ($L_{vis}^{self}$ and $L_{text}^{self}$), ensuring the model understands fundamental cross-modal relationships.
- Task-Specific Evaluation: Introduces a classification objective ($L_{comp}^{cls}$) to begin distinguishing between pristine and falsified samples.
Phase 2: Knowledge Distillation
To refine performance, a larger "Teacher" network ($f_{te}$) is pretrained and frozen. A smaller "Student" network ($f_{st}$) is then fine-tuned to mimic the Teacher’s probability distribution queue ($p_i^{te}$) through a knowledge distillation loss ($L_{dis}^{self}$). This allows the Student to capture complex, domain-specific nuances that might be missed during standard training.
Experimental Results
Performance on NewsCLIPpings
SSDL was evaluated on the NewsCLIPpings Dataset, which features challenging splits based on semantic similarity, person matching, and scene context. On the comprehensive Balanced split, SSDL achieved a ~3% improvement in total accuracy compared to the CLIP baseline. Most notably, the model showed a 7% gain specifically in identifying "Falsified" samples, demonstrating superior sensitivity to subtle contextual mismatches.
Data Efficiency and Explainability
SSDL exhibits remarkable data efficiency, matching baseline performance while using only 50-75% of the training data. Beyond raw metrics, the framework utilizes LIME (Local Interpretable Model-Agnostic Explanations) to generate visual heatmaps. These segmentations highlight specific image regions—such as background objects or people—that contributed to a "Falsified" verdict, aiding human verification.
Conclusion
SSDL provides a robust, explainable approach to detecting out-of-context misinformation. By combining self-supervised alignment with knowledge distillation, the framework overcomes data scarcity and provides critical visual evidence for its decisions. Future research will explore extending this architecture to video-text content and integrating external world knowledge to detect even more subtle geographic or temporal inconsistencies.
Summary updated