Synchronizing Approach in Designing Annotation Guidelines for Multilingual Datasets: A COVID-19 Case Study Using English and Japanese Tweets

Kiki Ferawati, Wan Jou She, Shoko Wakamiya, Eiji AramakiACL 2024

TL;DR

This paper introduces a novel "synchronizing approach" to create consistent annotation guidelines for multilingual datasets, demonstrated through a COVID-19 mask discourse analysis of English and Japanese tweets. By iteratively refining guidelines based on annotator feedback and cross-lingual discussions, the authors achieved improved inter-annotator agreement, offering a practical method for cross-cultural NLP research.

What Problem Does This Paper Address?

Comparing societal attitudes across cultures is crucial, but obtaining comparable data from different languages presents a significant challenge, especially for tasks like sentiment or stance analysis. Existing methods often rely on machine translation, which can lose cultural nuances and lead to inaccurate interpretations. This paper tackles the problem of creating robust, comparable annotation guidelines for multilingual datasets, specifically focusing on the nuanced discourse surrounding COVID-19 mask-wearing in the U.S. and Japan, where cultural responses to public health measures differed significantly. The goal is to enable reliable cross-cultural analysis of social media data.

Methods

The core of this paper is the synchronizing approach for designing annotation guidelines. This iterative process involves:

  • Pilot Annotation: Initial annotation of a small sample of tweets in both English and Japanese using draft guidelines.
  • Guideline Translation and Refinement: Translating guidelines and then iteratively revising them based on annotator feedback and disagreements. This includes incorporating linguistic nuances and examples from both languages.
  • Synchronized Annotation Rounds: Conducting annotation in parallel for both languages, with regular discussions between authors and annotators to clarify ambiguities and resolve disagreements.
  • Inter-Annotator Agreement (IAA) Evaluation: Using Cohen's kappa to measure agreement at each stage and round, with a target of weak agreement (0.40-0.59) as acceptable.
  • Focus on Disagreements: Analyzing disputed tweets to identify whether disagreements stem from guideline ambiguity or genuine interpretation differences.

This method contrasts with simply translating existing guidelines, aiming instead for a truly synchronized understanding across languages and cultures.

Datasets

The study utilized a dataset of COVID-19 related tweets collected from January 1st, 2020, to December 31st, 2022, using the Twitter API.

  • Source: Tweets from the U.S. (English) and Japan (Japanese).
  • Size: Approximately 1,102,876 English tweets and 589,927 Japanese tweets were initially obtained.
  • Preprocessing: Tweets were filtered for geo-tagged location data (city-level), links were removed, and usernames were anonymized. A minimum character threshold was applied to filter out irrelevant short tweets (25 characters for English, 10 for Japanese).
  • Annotation Sample: A subset of tweets was used for the annotation process, with specific sample sizes for each round (10%, 50%, 100%).

This study did not introduce a new benchmark dataset but rather focused on the methodology for creating annotation guidelines for such data.

Key Insights and Ideas

  • Cultural Nuances Matter: Direct translation of guidelines is insufficient; understanding and incorporating cultural context and linguistic expressions from both languages is critical for effective multilingual annotation.
  • Iterative Refinement is Key: A synchronized, iterative process of annotation, discussion, and guideline revision significantly improves inter-annotator agreement, even for challenging tasks.
  • Sarcasm and Ambiguity are Major Hurdles: Tweets containing sarcasm and ambiguous expressions, especially those influenced by cultural communication styles (e.g., indirectness in Japanese), pose significant challenges for annotators and require careful guideline development.
  • Stance and Behavior are Difficult: Identifying a user's stance on mask-wearing and their actual mask-wearing behavior from tweets proved to be the most challenging annotation tasks, requiring more refined guidelines and yielding lower agreement scores.
  • Dichotomous Choices Can Be Necessary: Due to the inherent ambiguity in some tweets, the authors opted for dichotomous choices (e.g., "against" vs. "not against") for stance, omitting a "neutral" option to simplify interpretation and improve agreement.

Key Results

  • Improved IAA: The synchronizing approach led to a consistent increase in inter-annotator agreement across annotation rounds for both English and Japanese tweets.
  • Relevancy Stage: Agreement scores for tweet relevancy improved significantly, reaching moderate agreement levels (e.g., final 0.79 for English, 0.92 for Japanese).
  • Stance Stage: Agreement for stance remained in the weak agreement range throughout the process (e.g., final 0.46 for English, 0.59 for Japanese), indicating the inherent difficulty of this task.
  • Mask-Wearing Stage: This stage showed notable improvement, moving from very low agreement in early rounds (e.g., 0.21 for English) to desirable weak agreement by the end (e.g., final 0.55 for English, 0.47 for Japanese).
  • Cross-Lingual Consistency: The synchronized approach helped achieve comparable agreement scores for both languages, demonstrating the effectiveness of the method.

Limitations and Future Work

  • Language Limitation: The study is limited to English and Japanese. Future work could extend this approach to other language pairs.
  • Tweet Specificity: The focus on short, informal text (tweets) means the findings might not directly generalize to other text types.
  • Dichotomous Choices: The decision to omit a "neutral" stance category, while aiding agreement, might limit the granularity of analysis. Future work could explore methods to handle neutral stances more effectively.
  • Geographic/Demographic Bias: The use of geo-tagged tweets means the sample is limited to users who provide location information, potentially introducing bias.
  • Future Work: The authors suggest that the developed multilingual guidelines can be used to obtain more annotated samples for future comparative studies on mask opinions and other COVID-19 related topics. Further research could explore how to better handle sarcasm and ambiguous expressions, and how cultural differences impact the interpretation of stance.

Summary updated

BibTeX
@inproceedings{ferawati2024synchronizing,
  author = {Kiki Ferawati and Wan Jou She and Shoko Wakamiya and Eiji Aramaki},
  title = {{Synchronizing Approach in Designing Annotation Guidelines for Multilingual Datasets: A COVID-19 Case Study Using English and Japanese Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure