Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

Giscard Biamby, Grace Luo, Trevor Darrell, Anna Rohrbach

Abstract

Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.

Fighting "Cheap Fakes" with Multimodal AI

In the battle against misinformation, the focus often lands on high-tech "deepfakes." However, a much more common and insidious threat is the "cheap fake"—a perfectly real image paired with a completely false caption to change its context. Researchers from UC Berkeley have introduced Twitter-COMMs, a massive dataset and a new detection framework designed to catch these out-of-context (OOC) media posts in high-stakes domains: Climate Change, COVID-19, and Military Vehicles.

The Problem: Out-of-Context (OOC) Misinformation

The primary challenge is OOC misinformation, where "pristine" images are paired with "falsified" text. Unlike deepfakes, the pixels are not manipulated; instead, the narrative is hijacked. Existing datasets for this problem are often too small or focus only on text-based claims. Furthermore, most models struggle with "hard" mismatches—where the fake caption is topically related to the image but describes a different specific event or entity.

An example of a 'cheap fake' where the text describes an F-35 but the image shows a Rafale fighter jet, representing a 'hard negative' mismatch.

The Twitter-COMMs Dataset

The core contribution is the Twitter-COMMs Dataset, consisting of 884,331 unique tweets collected via the Twitter API. The distribution covers three critical areas: 64.5% COVID-19, 24% Climate Change, and 11.5% Military Vehicles. The training set includes approximately 2.4 million samples, categorized into "Pristine" (real), "Random" (easy fakes), and "Hard" (difficult fakes) pairs.

Methodology: CLIP and Hard Negative Mining

The researchers proposed a detection method based on CLIP (Contrastive Language-Image Pre-training), utilizing three key strategies:

  1. Multimodal Fusion: Instead of simple concatenation, they used an element-wise product of CLIP embeddings. This "Multiply" strategy encourages the model to find specific interactions between modalities, leading to a 7% performance boost.
  2. Hard Negative Mining: To make the model robust, they trained it on "hard" mismatches—images that are textually similar to a caption but do not match it. This forces the model to learn fine-grained semantic inconsistencies rather than just identifying broad topic mismatches.
  3. Joint Training: A single model trained on all three topics performed as well as or better than specialized "expert" models.

Key Insights and Results

One of the most significant findings was the role of OCR as a Truth Signal. The researchers found that images containing text, such as infographics or news headers, are significantly easier to verify. When an English OCR model could read text within the image, detection performance jumped by over 6%.

An infographic from the COVID-19 domain where embedded text provides strong signals for verification via OCR.

On human-generated test data mimicking real-world misinformation, the model improved the Probability of Detection (pD) by 11% at a low False Alarm Rate (0.1 FAR). Even though the model was trained on synthetically generated fakes, it generalized remarkably well to "in-the-wild" style fakes created by humans.

Limitations and Future Work

While the model excels in the three chosen domains, it still faces challenges with "within-cluster" fakes—extremely similar swaps, such as two different models of the same aircraft. Future work may need to incorporate external knowledge bases or more granular entity recognition to bridge this gap and address inherited biases from the underlying CLIP model.

Summary updated

Abstract

Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.
BibTeX
@inproceedings{biamby2022twitter,
  author = {Giscard Biamby and Grace Luo and Trevor Darrell and Anna Rohrbach},
  title = {{Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.110},
}
Figure