How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

In the split second it takes to scroll past a news story on social media, two things grab your attention: the headline and the thumbnail. While significant research has focused on developing AI to detect misinformation in text, the visual bait—often used as a deceptive hook—is frequently overlooked. Researchers from Soongsil University and Adobe Research have investigated a specific tactic of misinformation: the use of "unrepresentative" thumbnails—images that have little to nothing to do with the actual story.

The Problem of Visual Clickbait

Most social media users consume news via "snippets" on platforms like X (formerly Twitter) or Facebook. Studies indicate that many users share or form opinions based on these previews without ever clicking the link. Fake news creators exploit this behavior by using sensational or emotionally charged thumbnails that do not represent the article's content—a visual form of clickbait.

The technical challenge in identifying this tactic lies in measuring "semantic incongruity." To detect it, a model must understand both the nuances of a news headline and the visual content of an image, then determine if they are contextually related. This research fills a critical gap by focusing on the multimodal relationship between the headline and the thumbnail.

Methodology: Leveraging CLIP

The researchers utilized OpenAI’s CLIP (Contrastive Language-Image Pre-training), a state-of-the-art model trained on millions of image-text pairs to map both modalities into a single shared embedding space.

The CLIP-based framework calculates the cosine similarity (CLIPScore) between the text embedding (c) and image embedding (v) to measure semantic relevance.

As shown in the figure above, the core metric is the CLIPScore: the cosine similarity between the vector representation of a news title ($c$) and its thumbnail image ($v$). A high score indicates the image and text are closely related, while a low score suggests a mismatch.

Beyond simple scoring, the authors developed a CLIP-classifier (a Multi-Layer Perceptron) trained to distinguish between "congruent" pairs (original headline and image) and "incongruent" pairs (where images were randomly swapped). They compared this approach against ViLT (Vision-and-Language Transformer) as a baseline.

Datasets and Robustness

The study analyzed news articles shared on Twitter between January and August 2021, comparing five "trustworthy" outlets (e.g., Reuters, The Guardian) against four "fake news" outlets (e.g., Activist Post, WorldNetDaily). To ensure robust results, they created three subsets:

  • Whole: The entire raw dataset of 109,715 articles.
  • COVID: Articles specifically containing COVID-19 keywords.
  • COVID-wo-faces: A subset where images containing human faces were removed. This was necessary because CLIP occasionally struggles to link specific, non-famous individuals mentioned in text to their faces in images.

Key Insights and Results

The "Incongruity Signature"

The most striking finding is that fake news sources consistently use images that are less relevant to their titles compared to legitimate news. This "incongruity signature" was statistically significant across all tested datasets ($p < 0.001$). The "medium" effect size (Cohen’s d ≈ 0.6) suggests that image-text mismatch is a reliable marker of unreliable news sources.

Detection Performance

The CLIPScore (zero-shot) method achieved an impressive 0.934 accuracy and 0.984 AUROC on the test set, vastly outperforming the ViLT baseline (0.601 accuracy). In a manual evaluation of the top-30 articles flagged as "most incongruent" by the model, CLIPScore achieved a precision of 0.87, demonstrating its effectiveness in identifying misleading thumbnails in real-world scenarios.

Limitations and Future Directions

While the results are promising, the study notes several areas for future work:

  • Source Diversity: Expanding the analysis beyond the initial nine news outlets to a broader range of international and local sources.
  • Title vs. Body: Using the news title as a proxy for content may miss instances where a thumbnail represents a specific detail in the article body. Future models could incorporate the full article text.
  • The Satire Problem: Distinguishing between malicious misinformation and intentional satire, where incongruity is often a stylistic choice, remains a challenge for multimodal AI.

By proving that the "thumbnail-text gap" is a measurable characteristic of fake news, this research provides a foundation for more robust, multimodal fact-checking tools that evaluate the entire news snippet rather than just the text.

Summary updated

Abstract

This study investigates how fake news uses a thumbnail for a news article with a focus on whether a news article's thumbnail represents the news content correctly. A news article shared with an irrelevant thumbnail can mislead readers into having a wrong impression of the issue, especially in social media environments where users are less likely to click the link and consume the entire content. We propose to capture the degree of semantic incongruity in the multimodal relation by using the pretrained CLIP representation. From a source-level analysis, we found that fake news employs a more incongruous image to the main content than general news. Going further, we attempted to detect news articles with image-text incongruity. Evaluation experiments suggest that CLIP-based methods can successfully detect news articles in which the thumbnail is semantically irrelevant to news text. This study contributes to the research by providing a novel view on tackling online fake news and misinformation. Code and datasets are available at https://github.com/ssu-humane/fake-news-thumbnail.
BibTeX
@misc{choi2022how,
  author = {Hyewon Choi and Yejun Yoon and Seunghyun Yoon and Kunwoo Park},
  title = {{How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image}},
  year = {2022},
  eprint = {2204.05533},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.05533},
}
Figure