How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

TL;DR

Researchers have found that fake news outlets frequently use "unrepresentative" thumbnails—images that have little to do with the actual headline—to mislead social media users who skim content without clicking. By leveraging OpenAI’s CLIP model to measure the semantic "distance" between headlines and images, the authors developed a method that detects these incongruous pairs with over 93% accuracy, offering a new multimodal lens for identifying misinformation.

What Problem Does This Paper Address?

On social media, the vast majority of users consume news through "snippets"—the combination of a headline and a thumbnail image. Previous research shows that many users never actually click the link to read the full article, meaning their entire impression of an issue is formed by these two elements.

While much work has gone into analyzing the text of fake news (looking for "clickbait" patterns or linguistic markers), the relationship between the text and the accompanying image has been under-explored. The authors hypothesize that low-credibility sources are more likely to use semantically irrelevant or "incongruous" images to grab attention or promote false beliefs. The challenge lies in automatically measuring "relevance" across two different types of data: text and vision.

Methods

The core of the paper’s approach is CLIP (Contrastive Language-Image Pre-training). CLIP is a multimodal model trained on hundreds of millions of image-caption pairs, allowing it to project both images and text into the same high-dimensional "embedding space."

  1. CLIPScore: The authors calculate the cosine similarity between the vector representation of a news title ($c$) and its thumbnail image ($v$). A high CLIPScore indicates the image and text are semantically aligned; a low score suggests they are irrelevant to one another.
  2. CLIP-classifier: Beyond simple similarity, the authors built a neural network (a multilayer perceptron) that takes the CLIP embeddings as input to classify whether a pair is "congruent" or "incongruent."
  3. Baseline Comparison: They compared their CLIP-based approach against ViLT (Vision-and-Language Transformer), another state-of-the-art multimodal model, to see which was better at capturing news-specific nuances.

Datasets

The authors collected a massive dataset of 109,715 news articles shared on Twitter between January and September 2021, spanning five "trustworthy" sources (e.g., Reuters, The Guardian) and four "fake" sources (e.g., Activist Post, WorldNetDaily). They created three specific subsets:

  • Whole: The full dataset of all collected articles.
  • COVID: Articles specifically containing COVID-19 keywords (to test performance on a specific, high-stakes topic).
  • COVID-wo-faces: A subset of the COVID data where images containing human faces were removed. This was done because CLIP is known to struggle with recognizing specific individuals who aren't global celebrities, which could lead to "false negatives" in similarity scoring.

Key Insights and Ideas

  • The "Incongruity Gap": The most significant finding is that fake news outlets consistently show lower semantic similarity between titles and thumbnails compared to general news. This confirms that visual-textual mismatch is a measurable characteristic of misinformation.
  • Visual Baiting: Fake news doesn't just use "fake" images; it uses irrelevant ones. This suggests a strategy of using provocative but unrelated visuals to drive engagement, knowing that the lack of relevance won't be noticed by users who don't click through.
  • CLIP's "Face" Problem: The researchers noted that CLIP often fails to connect a person's name in a headline to their face in an image unless they are extremely famous. By creating the "wo-faces" (without faces) dataset, they demonstrated that the model's detection accuracy improves when it focuses on objects and scenes rather than specific identities.

Key Results

  • Statistical Significance: Using Cumulative Distribution Functions (CDFs), the authors showed a clear, statistically significant difference in CLIPScores between general and fake news across all datasets (Cohen’s d effect sizes between 0.54 and 0.59).
  • Detection Performance: The CLIPScore (zero-shot) method achieved an accuracy of 93.4% and an AUROC of 0.984 on the test set, vastly outperforming the ViLT baseline (60.1% accuracy).
  • Human Validation: In a real-world test on "fake news" samples, the CLIP-based detection reached a precision of 0.87 for the top-30 most "incongruous" articles identified by the model, as verified by human annotators.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Source Diversity: The study only looked at nine news outlets. While balanced for political bias, a larger variety of sources would make the findings more robust.
  • The "Face" Limitation: As noted, the model struggles with proper nouns and faces. Future work could integrate facial recognition or named entity linking to bridge this gap.
  • Title vs. Body: The study used the news title as a proxy for the content. However, some thumbnails might be relevant to the body text but not the headline. Future iterations could incorporate the full article text for a more holistic "relevance" check.
  • CLIP Biases: Since CLIP was trained on web data, it may carry inherent biases that affect how it perceives "relevance" in different cultural or social contexts.

Summary updated

Abstract

This study investigates how fake news uses a thumbnail for a news article with a focus on whether a news article's thumbnail represents the news content correctly. A news article shared with an irrelevant thumbnail can mislead readers into having a wrong impression of the issue, especially in social media environments where users are less likely to click the link and consume the entire content. We propose to capture the degree of semantic incongruity in the multimodal relation by using the pretrained CLIP representation. From a source-level analysis, we found that fake news employs a more incongruous image to the main content than general news. Going further, we attempted to detect news articles with imagetext incongruity. Evaluation experiments suggest that CLIP-based methods can successfully detect news articles in which the thumbnail is semantically irrelevant to news text. This study contributes to the research by providing a novel view on tackling online fake news and misinformation.
BibTeX
@misc{choi2022how,
  author = {Hyewon Choi and Yejun Yoon and Seunghyun Yoon and Kunwoo Park},
  title = {{How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image}},
  year = {2022},
  eprint = {2204.05533},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.05533},
}
Figure