From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media

Katharina Prasse, Steffen Jung, Isaac Bravo, Stefanie Walter, Patrick Knab, Christian Bartelt, Margret KeuperarXiv 2026 · arXiv

TL;DR

While social media is flooded with visual climate change communication, analyzing millions of images has traditionally been a manual, cost-prohibitive bottleneck for researchers. This paper systematically benchmarks vision-language models (VLMs) against traditional social science codebooks, revealing that while per-image accuracy is only moderate, VLMs excel at accurately recovering aggregate, population-level discourse trends.

What Problem Does This Paper Address?

Most computational social science research on climate change focuses heavily on text (tweets, news articles, sentiment analysis). However, social media is inherently visual—driven by memes, infographics, protest photography, and wildlife imagery that profoundly shape public perception.

Manually annotating these images at scale is impossible (X/Twitter alone sees over 500,000 climate-related images annually). While computer vision models offer a scalable alternative, we lack systematic frameworks to understand how well modern VLMs align with theoretically grounded, nuanced social science taxonomies. The authors tackle this gap by evaluating model selection, prompt engineering, and taxonomy design for automated visual discourse analysis.

Methods

The authors establish a comprehensive evaluation pipeline comparing two main paradigms:

  1. Promptable VLMs: Six models were tested, including open-weight options (Qwen3-VL-8B/30B, Moondream3, Gemma-3-4B) and closed-weight APIs (GPT-5.4-mini, Gemini-3.1-flash-lite).
  2. Zero-shot CLIP-like models: 15 different configurations from the OpenCLIP library.

The core methodological innovation is the shift from purely instance-level evaluation (e.g., F1-score, accuracy) to distributional evaluation using metrics like Jensen-Shannon (JS) divergence and Total Variation. This assesses whether a model can accurately recreate the overall "pie chart" of a discourse topic across a massive dataset, even if it occasionally misclassifies individual images. Additionally, they systematically ablated prompt components, testing the impact of category explanations, super-category definitions, spatial localization instructions, and chain-of-thought (CoT) reasoning.

Datasets

The models were evaluated on two newly curated datasets spanning five annotation dimensions (animals, climate action, consequences, image setting, and image type):

  • ClimateCT (1,038 images): A high-quality, curated dataset of popular tweets, manually annotated by independent domain experts (social scientists).
  • ClimateTV (1,220,784 images): A massive, heterogeneous corpus representing the long tail of climate images on X from 2019–2022. To validate this dataset, the authors used a two-stage VLM pipeline to generate labels and then manually validated 50,000 of them (yielding 250,000 discrete image-label pairs) via crowd workers.

Key Insights and Ideas

  • Trends Over Instances: The standout "aha" moment is that VLMs can reliably recover population-level discourse trends (JS divergence $\le 0.01$ for categories like animals and type) even when per-image accuracy is sub-optimal. Errors are largely non-systematic, meaning they wash out at scale.
  • Chain-of-Thought (CoT) Backfires: Counterintuitively, explicitly triggering CoT reasoning (e.g., "think carefully step by step") reduced performance for most visual classification tasks. For well-defined visual categories, extra reasoning steps simply introduced noise.
  • Asymmetric Confusion: Models don't fail randomly; they fail hierarchically. For example, pets are frequently misclassified as farm animals, and polar bears are misclassified as land mammals, but rarely vice versa. Models inherently default to broader, more frequently seen categories when confused.
  • Size Isn't Everything: Smaller open-weight models like Qwen3-8B performed on par with or better than closed models like GPT-5.4-mini, offering a highly cost-effective alternative for researchers running inference on millions of images.

Key Results

  • Top Performer: Gemini-3.1-flash-lite consistently outperformed all other models across both datasets and all super-categories, achieving a macro accuracy of 0.76 on animals and 0.70 on climate action.
  • VLM vs. CLIP Gap: Traditional CLIP-like models severely struggled, scoring up to 0.29 points lower in macro accuracy compared to promptable VLMs. CLIP models particularly failed at identifying negative classes (e.g., classifying an image as having "no climate consequences").
  • Prompting Specifics: The optimal prompt style depends entirely on the abstraction of the category. Simple, visually self-explanatory categories (image type) perform best with short prompts containing only class names. Abstract categories (climate action, consequences) see an accuracy boost of up to +0.3 when provided with extended explanations and examples.

Limitations and Future Work

The study is restricted to data from a single platform (X/Twitter), leaving the generalizability to highly visual platforms like Instagram or TikTok unproven. Furthermore, the annotation scheme focuses on structural elements and omits complex psychological dimensions like emotional affect, political stance, or narrative framing. The authors suggest that future work should focus on developing hybrid human-machine validation pipelines to safely navigate the systematic biases introduced by automated taxonomies.

Summary updated

Abstract

Social media platforms have become primary arenas for climate communication, generating millions of images and posts that - if systematically analysed - can reveal which communication strategies mobilise public concern and which fall flat. We aim to facilitate such research by analysing how computer vision methods can be used for social media discourse analysis. This analysis includes application-based taxonomy design, model selection, prompt engineering, and validation. We benchmark six promptable vision-language models and 15 zero-shot CLIP-like models on two datasets from X (formerly Twitter) - a 1,038-image expert-annotated set and a larger corpus of over 1.2 million images, with 50,000 labels manually validated - spanning five annotation dimensions: animal content, climate change consequences, climate action, image setting, and image type. Among the models benchmarked, Gemini-3.1-flash-lite outperforms all others across all super-categories and both datasets, while the gap to open-weight models of moderate size remains relatively small. Beyond instance-level metrics, we advocate for distributional evaluation: VLM predictions can reliably recover population level trends even when per-image accuracy is moderate, making them a viable starting point for discourse analysis at scale. We find that chain-of-thought reasoning reduces rather than improves performance, and that annotation dimension specific prompt design improves performance. We release tweet IDs and labels along with our code at https://github.com/KathPra/Codebooks2VLMs.git.
BibTeX
@misc{prasse2026from,
  author = {Katharina Prasse and Steffen Jung and Isaac Bravo and Stefanie Walter and Patrick Knab and Christian Bartelt and Margret Keuper},
  title = {{From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media}},
  year = {2026},
  eprint = {2604.21786},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.21786},
}

Loading related content...

Figure