TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore

Nirmalendu Prakash, Ming Shan Hee, Roy Ka-Wei Lee

TL;DR

This paper introduces TotalDefMeme, a novel multimodal, multi-attribute meme dataset focused on Singapore's Total Defence policy. It aims to bridge the gap in understanding public sentiment towards defense policies, which traditional surveys often miss, by leveraging the expressive nature of internet memes.

What Problem Does This Paper Address?

Understanding public sentiment towards government policies is crucial, but traditional methods like surveys can be limited. Social media, particularly internet memes, offers a more unfiltered and nuanced perspective on public opinion. However, existing meme datasets often focus on negative aspects like hate speech and are primarily based on Western cultural contexts. This paper addresses the lack of datasets that capture public sentiment towards defense policies, specifically Singapore's Total Defence, and the need for multimodal analysis that considers both visual and textual elements of memes within a specific cultural context.

Methods

The authors propose a three-phase pipeline for constructing the TotalDefMeme dataset:

  1. Dataset Collection: They employed a keyword-based approach using Google Search and scraped publicly available social media platforms (Reddit, Instagram) to gather memes related to Singapore's Total Defence. Keywords were crafted based on the policy's definitions.
  2. Dataset Processing: Memes were filtered for quality (resolution, text length), text was extracted using EasyOCR, duplicates were removed using pHash, and memes without text were discarded. This resulted in a final dataset of 5,301 memes.
  3. Dataset Annotation: Six annotators familiar with Singaporean culture and Total Defence concepts were recruited. They identified Singapore-related memes and annotated them with:
  4. Meme Type: Generic vs. Singapore-related.
  5. Defence Pillars: Military, Civil, Economic, Social, Psychological, Digital, or Others.
  6. Topic Tags: Free-text tags capturing essential elements.
  7. Stances: Support, Against, or Neutral towards the identified pillars. Quality control involved double annotation and a third annotator for disagreements.

Datasets

The paper introduces the TotalDefMeme dataset, a large-scale, multimodal, and multi-attribute meme dataset specifically designed to capture public sentiments towards Singapore's Total Defence policy. It contains 5,301 memes, annotated with meme type, defence pillars, topic tags, and stances. This dataset differs from existing ones by focusing on a public policy domain and incorporating a specific cultural context (Singapore/Singlish).

Key Insights and Ideas

  • Memes as a Policy Sentiment Gauge: The paper highlights that memes can offer a more authentic reflection of public sentiment towards policies than traditional surveys, especially for nuanced or potentially sensitive topics like defense.
  • Cultural Specificity Matters: The dataset's focus on Singapore and its colloquial language (Singlish) is crucial for understanding local interpretations and sentiments, which generic datasets might miss.
  • Multimodality is Key: Analyzing memes requires understanding the interplay between visual and textual elements, a challenge that TotalDefMeme is designed to facilitate.
  • Class Imbalance in Defence Pillars: The dataset exhibits significant class imbalance across the Total Defence pillars, with "Military" being dominant and "Digital" and "Social" being underrepresented. This presents a challenge and an opportunity for developing robust models.
  • Subjectivity in Stance Annotation: The lower inter-annotator agreement for stance classification (Krippendorff's Alpha of 0.21) underscores the inherent subjectivity in interpreting sentiment from memes.

Key Results

  • The TotalDefMeme dataset contains 5,301 memes, with a significant portion (more than half) identified as Singapore-related.
  • Inter-annotator agreement scores (Krippendorff's Alpha) were moderate for meme type (0.65) and defence pillars (0.55), indicating good annotation quality, but weaker for stances (0.21).
  • Baseline experiments for meme clustering showed that multimodal embeddings (VisualBERT, CLIP) performed better than unimodal ones, with VisualBERT achieving the highest Silhouette score (0.094).
  • For aspect-based stance classification, the CLIP model achieved the best accuracy, with scores of 0.57 for pillar classification and 0.54 for stance classification.

Limitations and Future Work

  • Class Imbalance: The dataset suffers from class imbalance in defence pillar annotations, which can affect model performance.
  • Stance Annotation Subjectivity: The lower agreement for stance annotation highlights the challenge of subjective interpretation.
  • Future Work: The authors suggest several interdisciplinary research directions:
  • Analysis and assessment of country's total defence readiness.
  • Multimodal aspect-based stance classification.
  • Multimodal meme clustering.
  • Domain adaptation of meme analysis techniques.
  • Applying the annotation framework to other countries' defense policies.

Summary updated

Abstract

Total Defence is a defence policy combining and extending the concept of military defence and civil defence. While several countries have adopted total defence as their defence policy, very few studies have investigated its effectiveness. With the rapid proliferation of social media and digitalisation, many social studies have been focused on investigating policy effectiveness through specially curated surveys and questionnaires either through digital media or traditional forms. However, such references may not truly reflect the underlying sentiments about the target policies or initiatives of interest. People are more likely to express their sentiment using communication mediums such as starting topic thread on forums or sharing memes on social media. Using Singapore as a case reference, this study aims to address this research gap by proposing TotalDefMeme, a large-scale multi-modal and multi-attribute meme dataset that captures public sentiments toward Singapore's Total Defence policy. Besides supporting social informatics and public policy analysis of the Total Defence policy, TotalDefMeme can also support many downstream multi-modal machine learning tasks, such as aspect-based stance classification and multi-modal meme clustering. We perform baseline machine learning experiments on TotalDefMeme and evaluate its technical validity, and present possible future interdisciplinary research directions and application scenarios using the dataset as a baseline.
BibTeX
@inproceedings{prakash2023totaldefmeme,
  author = {Nirmalendu Prakash and Ming Shan Hee and Roy Ka-Wei Lee},
  title = {{TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore}},
  booktitle = {MMSys},
  year = {2023},
  doi = {10.1145/3587819.3592545},
}
Figure