TL;DR
This paper investigates how social, cultural, and political (SCP) symbols in Instagram images influence user engagement and content spread during anti-disinformation campaigns. By combining advanced LLMs for symbol extraction and an epidemiological model (SEIZ) for spread analysis, the study finds that posts with more SCP symbols and better text-image alignment lead to higher engagement and faster dissemination.
What Problem Does This Paper Address?
In the age of visually-driven social media like Instagram, images are a primary medium for communication. However, simply analyzing the literal content of images is insufficient to grasp their full impact, especially in complex contexts like political campaigns. Misinformation often leverages subtle visual cues and cultural symbols to influence public opinion. This paper addresses the challenge of understanding how these visual elements, specifically social, cultural, and political (SCP) symbols, contribute to user engagement (likes) and the spread of information (dissemination) in anti-disinformation campaigns. It seeks to bridge the gap between visual content and its reception by users, recognizing that user-generated text often doesn't fully align with accompanying images.
Methods
The study employs a multi-faceted approach:
- SCP Symbol Extraction: Advanced Large Language Models (LLMs), specifically GPT-4 Omni and Gemini Pro Vision, were used to identify and categorize social, cultural, and political symbols within Instagram images. This involved a detailed prompt designed to extract these symbols and provide reasoning. The performance of both LLMs was compared.
- Semantic Similarity Analysis: To assess text-image alignment, the researchers calculated the semantic similarity between SCP symbols identified in user-generated text and the descriptions generated by LLMs for the same images. The Universal Sentence Encoder (USE) was used to create embeddings for this analysis.
- Epidemiological Modeling: The SEIZ (Susceptible, Exposed, Infectious, Skeptical) model, a compartmental model adapted from public health, was used to simulate and analyze the spread of content. This model tracks users through different states of engagement (from exposure to liking/sharing). The model was fitted to Instagram data to estimate infection rates and dissemination speed.
Datasets
The study utilized a dataset of 3,097 Instagram images collected using the Apify tool. These images were related to keywords such as "DPP," "Taiwan Elections," "Huang Guochang," "Taiwan People Party," and "Taiwan Presidential Election," specifically focusing on the Taiwan 2024 Election Anti-Disinformation Campaign. The dataset was curated through expert selection and snowball sampling.
Key Insights and Ideas
- Visuals Matter, Especially When Rich in Symbols: Posts featuring a greater number of social, cultural, and political symbols consistently attracted more likes, indicating that these elements significantly enhance user engagement.
- Text-Image Alignment Amplifies Impact: When the SCP symbols in user-generated text closely align with the LLM-generated descriptions of the images (high semantic similarity), user engagement increases. This suggests that a coherent narrative across modalities is more persuasive.
- Symbolic Content Spreads Faster: The SEIZ model revealed that posts with more SCP symbols and higher text-image similarity spread more rapidly, exhibiting higher "infection rates," which is crucial for effective anti-disinformation campaigns.
- LLMs as Powerful Analytical Tools: The study demonstrates the effectiveness of LLMs like GPT-4 Omni and Gemini Pro Vision in not only extracting complex visual information but also in facilitating multimodal analysis, offering a robust framework for future research.
- Interdisciplinary Approach is Key: Combining semiotics, LLMs, and epidemiological modeling provides a powerful lens for understanding online information dynamics, moving beyond traditional single-modality analyses.
Key Results
- Symbol Impact: Posts with all three types of SCP symbols (Category 4) showed the highest engagement (likes). Conversely, posts with no symbols (Category 1) had the lowest engagement.
- LLM Performance: Gemini Pro Vision identified more social symbols, while GPT-40 identified more cultural symbols. Both models showed strong capabilities, with Gemini achieving a higher maximum similarity score (0.81) compared to GPT-40 (0.6124) in text-image alignment.
- Semantic Similarity and Engagement: A positive correlation was found between semantic similarity scores and user engagement (likes). Posts with higher alignment between user text and LLM descriptions received more likes.
- Dissemination Speed: The SEIZ model fitting indicated that posts with all SCP symbols had a near-perfect correlation (0.9960) and the highest infection rate (0.3), signifying rapid and extensive spread. Similarly, high text-image similarity (Range 3) resulted in a high correlation (0.9957) and a low relative error (0.0047), indicating effective modeling of rapid dissemination.
Limitations and Future Work
- LLM Limitations: The authors acknowledge that LLMs can be prone to errors, biases, and hallucinations, which can affect SCP symbol extraction.
- Population Specificity: The epidemiological model's findings are specific to the studied population in Taiwan and may not generalize to other contexts.
- Future Directions:
- Analyze liked comments to identify engagement patterns.
- Expand research to multiple platforms and explore video content.
- Develop methods to analyze videos with SCP symbols.
- Examine network structures to improve anti-disinformation strategies.
- Develop more sophisticated prompts by analyzing LLM dynamics.
Summary updated