Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

Giscard Biamby, Grace Luo, Trevor Darrell, Anna Rohrbach

Researchers at UC Berkeley have introduced Twitter-COMMs, a massive dataset of over 884,000 tweets designed to train models to detect "out-of-context" misinformation—where a real image is repurposed with a misleading caption. By fine-tuning a CLIP-based model using "hard" semantic mismatches, they achieved an 11% improvement in detecting human-generated fakes compared to strong baselines.

The Challenge of "Cheap Fakes"

Misinformation isn't always a high-tech "deepfake." Often, it is a "cheap fake": a legitimate, unedited image paired with a completely false caption to support a specific narrative. This is known as out-of-context (OOC) media.

An example of out-of-context misinformation where a caption about polar bears is paired with an unrelated image of traffic.

While OOC media is easy to produce, it is difficult to detect at scale because it requires a model to understand the nuanced semantic relationship between an image and its text. Most existing datasets for this task are either too small or lack the topical focus necessary for high-stakes domains. This research specifically targets three areas of significant public interest and national security: Climate Change, COVID-19, and Military Vehicles.

The Twitter-COMMs Dataset

The core contribution is the Twitter-COMMs Dataset, which includes 884,331 unique tweets collected via the Twitter API. The topical breakdown consists of 64.5% COVID-19, 24% Climate Change, and 11.5% Military Vehicles.

Examples of pristine, random, and hard negative samples across the three domains in the Twitter-COMMs dataset.

To make the dataset robust for training, the researchers augmented it to include 2.4 million samples, balancing "pristine" (real) tweets with both random and hard-negative mismatches. Training a model on random image-text pairs is often too simple; the model quickly learns to distinguish a "dog" caption from a "car" image. To address this, the authors generated "hard negatives"—falsified samples where the mismatched image is semantically similar to the original (e.g., swapping one type of military tank for another).

Methodology: Leveraging CLIP and Hard Negatives

The authors propose a detection method based on CLIP (Contrastive Language-Image Pre-training), a state-of-the-art model that maps images and text into a shared embedding space. Their approach involved several key strategies:

  • Multimodal Fusion: Instead of just concatenating image and text vectors, the researchers found that an element-wise product (Multiply) of the CLIP embeddings performed best. This "incentivizes" the classifier to stay faithful to the pre-initialized joint embedding space, leading to a 7% performance boost over simple concatenation.
  • Fine-tuning Strategy: They fine-tuned the upper layers of the CLIP backbone (RN50x16) while keeping lower layers frozen to maintain stability and memory efficiency.
  • Entity over Topic: Hard negatives force the model to look beyond broad topics. For example, a "hard" COVID-19 fake might swap a photo of one health official for another, requiring the model to recognize specific entities rather than just "medical-looking" imagery.

Key Results and Insights

The model, trained on synthetic data, generalized remarkably well to human-generated fakes. On human-curated sets (hTwitter), the proposed model achieved an 11% improvement in the Probability of Detection (pD) at a 0.1 False Alarm Rate (FAR) compared to a CLIP Zero-Shot baseline, reaching a balanced classification accuracy of 87.7%.

One interesting finding was the "OCR" Advantage: the model performs significantly better (over 6% improvement) on images that contain text, such as infographics or news screenshots. This suggests the model effectively uses optical character recognition to find contradictions between the text in the image and the tweet's caption. Conversely, the "Military Vehicles" category proved the most challenging due to the high visual similarity between different vehicle models, making even "hard" negatives difficult to distinguish from pristine pairs.

Limitations and Future Work

While promising, the current model is limited to English-language tweets and relies entirely on the relationship between the provided image and text. It does not query external knowledge bases or fact-checking sites, which might be necessary to debunk more subtle, fact-heavy misinformation. Additionally, the authors acknowledge that the underlying CLIP model has known biases regarding race, age, and gender. Future work aims to incorporate external knowledge and expand the dataset to include more diverse languages and a wider array of high-stakes topics.

Summary updated

BibTeX
@inproceedings{biamby2022twitter,
  author = {Giscard Biamby and Grace Luo and Trevor Darrell and Anna Rohrbach},
  title = {{Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.110},
}
Figure