Exploring Artificial Image Generation for Stance Detection

Zhengkang Zhang, Zhongqing Wang, Guodong ZhouEMNLP 2025

TL;DR

This paper introduces a novel approach to stance detection by generating artificial images from text to capture implicit authorial stance. By leveraging a multimodal model that combines text and these generated images, the authors achieve improved stance detection accuracy, demonstrating the power of visual context in understanding nuanced opinions.

What Problem Does This Paper Address?

Stance detection, a crucial task in natural language processing, aims to identify an author's attitude towards a specific topic. While deep learning models have made significant strides, they often rely solely on textual information, which may not fully capture the subtle or implicit stances conveyed by an author. Existing multimodal approaches typically depend on user-generated images, which can be irrelevant, inconsistent, or simply absent, limiting their applicability. This paper addresses the challenge of incorporating visual information effectively and reliably into stance detection by proposing a method to generate relevant and consistent images from text itself.

Methods

The core of this work lies in a three-stage process:

  1. Candidate Image Generation: The authors employ a text-to-image model to generate multiple candidate images for each text. These images are guided by three key criteria:
  2. Relevance (CR): The image must be pertinent to the text's content.
  3. Target Consistency (CT): The image must accurately reflect the target of the stance.
  4. Stance Consistency (Cs): The image must visually convey the same stance as the original text. This generation process involves predicting stance and sentiment, enriching image prompts with this information, and then using a text-to-image model like Stable-Diffusion-3.

  5. Image Quality Assessment & Re-ranking: To select the best generated image, a comprehensive evaluation framework is introduced. This includes:

  6. Point-wise Metrics: * Relevance Score (SR): Measured using CLIP to assess text-image similarity.
  7. Target Consistency Score (ST): Evaluated using a Vision Language Model (VLM) and ChatGPT to check if the image and its caption align with the target.
  8. Stance Consistency Score (Ss): Similar to ST, but focuses on aligning the image's stance with the text's stance.
  9. Graph-based Re-ranking: A graph-based algorithm (inspired by PageRank) is used to collectively re-rank candidate images, considering both their individual scores and their semantic relationships with other candidate images. This aims to select the most optimal image that best represents the original text's core meaning.
  10. Multimodal Stance Detection: Finally, a multimodal stance detection model is employed. This model takes both the original text and the selected artificial image as input. It uses a Large Language Model (LLM) for text encoding and a Vision Transformer (ViT) for image encoding. The fused representations are then used to predict the stance.

Datasets

The experiments were conducted on two established stance detection datasets:

  • SemEval-16: Contains five targets (Atheism, Climate Change is a Real Concern, Feminist Movement, Hillary Clinton, and Legalization of Abortion).
  • Covid-19: Contains four targets (Face Masks, Fauci, School Closures, and Stay at Home Orders).

The paper notes that SemEval-16 does not have a pre-defined validation set, and they followed a standard procedure to partition it.

Key Insights and Ideas

  • Visualizing Implicit Stance: The most striking insight is that artificially generated images can effectively "visualize" implicit stances that might be missed by text-only models. This suggests that visual representation can add a crucial layer of understanding.
  • Beyond User-Generated Images: The paper successfully decouples multimodal stance detection from the reliance on potentially problematic user-generated images. By generating images, they gain control over the visual content's relevance and consistency.
  • The Power of Coherence: The three criteria (Relevance, Target Consistency, Stance Consistency) are not just checkboxes; they are fundamental to creating images that genuinely enhance stance detection. The results show that incorporating all these criteria leads to the best performance.
  • Graph-based Re-ranking is Key: Simply picking the image with the highest individual score (Relevance, Target, or Stance Consistency) is less effective than using a graph-based approach that considers the relationships between candidate images. This highlights the importance of a holistic evaluation.

Key Results

  • The proposed multimodal approach, using artificially generated images, significantly outperforms baseline text-only models on both SemEval-16 and Covid-19 datasets.
  • On SemEval-16, the authors' method achieved an Favg of 0.818, outperforming simple image selection strategies (e.g., CLIP-only max score at 0.806) and demonstrating the superiority of their graph-based re-ranking.
  • The inclusion of generated images led to notable improvements in correctly predicted instances across different stance categories, with relative improvements ranging from 3.8% to 12.6%.
  • The paper also shows that their approach is effective across different multimodal models, indicating its universality.

Limitations and Future Work

  • Computational Intensity: Generating and evaluating candidate images using large VLM models is computationally intensive, which is acknowledged as a limitation.
  • Ethical Considerations: The authors recognize that image generation models can inherit and amplify social biases, and generated content is an interpretation, not a factual record. They mitigate this by establishing clear generation guidelines and an evaluation framework.
  • "None" Stance Impact: The "None" stance category appears to be more negatively impacted by visual information, likely because images inherently convey a certain stance, making it harder to represent neutrality.
  • Future Work: The paper suggests further research into improving the efficiency of the generation and evaluation process and exploring more sophisticated multimodal architectures.

Summary updated

BibTeX
@inproceedings{zhang2025exploring,
  author = {Zhengkang Zhang and Zhongqing Wang and Guodong Zhou},
  title = {{Exploring Artificial Image Generation for Stance Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure