TL;DR
This paper introduces TotalDefMeme, a novel multimodal, multi-attribute meme dataset focused on Singapore's Total Defence policy. It aims to bridge the gap in understanding public sentiment towards defense policies, which traditional surveys often miss, by leveraging the expressive nature of internet memes.
What Problem Does This Paper Address?
Understanding public sentiment towards government policies is crucial, but traditional methods like surveys can be limited. Social media, particularly internet memes, offers a more unfiltered and nuanced perspective on public opinion. However, existing meme datasets often focus on negative aspects like hate speech and are primarily based on Western cultural contexts. This paper addresses the lack of datasets that capture public sentiment towards defense policies, specifically Singapore's Total Defence, and the need for multimodal analysis that considers both visual and textual elements of memes within a specific cultural context.
Methods
The authors propose a three-phase pipeline for constructing the TotalDefMeme dataset:
- Dataset Collection: They employed a keyword-based approach using Google Search and scraped publicly available social media platforms (Reddit, Instagram) to gather memes related to Singapore's Total Defence. Keywords were crafted based on the policy's definitions.
- Dataset Processing: Memes were filtered for quality (resolution, text length), text was extracted using EasyOCR, duplicates were removed using pHash, and memes without text were discarded. This resulted in a final dataset of 5,301 memes.
- Dataset Annotation: Six annotators familiar with Singaporean culture and Total Defence concepts were recruited. They identified Singapore-related memes and annotated them with:
- Meme Type: Generic vs. Singapore-related.
- Defence Pillars: Military, Civil, Economic, Social, Psychological, Digital, or Others.
- Topic Tags: Free-text tags capturing essential elements.
- Stances: Support, Against, or Neutral towards the identified pillars. Quality control involved double annotation and a third annotator for disagreements.
Datasets
The paper introduces the TotalDefMeme dataset, a large-scale, multimodal, and multi-attribute meme dataset specifically designed to capture public sentiments towards Singapore's Total Defence policy. It contains 5,301 memes, annotated with meme type, defence pillars, topic tags, and stances. This dataset differs from existing ones by focusing on a public policy domain and incorporating a specific cultural context (Singapore/Singlish).
Key Insights and Ideas
- Memes as a Policy Sentiment Gauge: The paper highlights that memes can offer a more authentic reflection of public sentiment towards policies than traditional surveys, especially for nuanced or potentially sensitive topics like defense.
- Cultural Specificity Matters: The dataset's focus on Singapore and its colloquial language (Singlish) is crucial for understanding local interpretations and sentiments, which generic datasets might miss.
- Multimodality is Key: Analyzing memes requires understanding the interplay between visual and textual elements, a challenge that TotalDefMeme is designed to facilitate.
- Class Imbalance in Defence Pillars: The dataset exhibits significant class imbalance across the Total Defence pillars, with "Military" being dominant and "Digital" and "Social" being underrepresented. This presents a challenge and an opportunity for developing robust models.
- Subjectivity in Stance Annotation: The lower inter-annotator agreement for stance classification (Krippendorff's Alpha of 0.21) underscores the inherent subjectivity in interpreting sentiment from memes.
Key Results
- The TotalDefMeme dataset contains 5,301 memes, with a significant portion (more than half) identified as Singapore-related.
- Inter-annotator agreement scores (Krippendorff's Alpha) were moderate for meme type (0.65) and defence pillars (0.55), indicating good annotation quality, but weaker for stances (0.21).
- Baseline experiments for meme clustering showed that multimodal embeddings (VisualBERT, CLIP) performed better than unimodal ones, with VisualBERT achieving the highest Silhouette score (0.094).
- For aspect-based stance classification, the CLIP model achieved the best accuracy, with scores of 0.57 for pillar classification and 0.54 for stance classification.
Limitations and Future Work
- Class Imbalance: The dataset suffers from class imbalance in defence pillar annotations, which can affect model performance.
- Stance Annotation Subjectivity: The lower agreement for stance annotation highlights the challenge of subjective interpretation.
- Future Work: The authors suggest several interdisciplinary research directions:
- Analysis and assessment of country's total defence readiness.
- Multimodal aspect-based stance classification.
- Multimodal meme clustering.
- Domain adaptation of meme analysis techniques.
- Applying the annotation framework to other countries' defense policies.
Summary updated