A Dataset for Semantic Role Labelling of Hindi-English Code-Mixed Tweets

Riya Pal, Dipti SharmaACL 2019

TL;DR

This paper introduces the first dataset for Semantic Role Labelling (SRL) of Hindi-English code-mixed tweets, a crucial step for understanding informal online communication. They developed a novel annotation scheme and a rule-based system that achieves high accuracy, demonstrating the feasibility of SRL for this challenging linguistic phenomenon.

What Problem Does This Paper Address?

Modern social media platforms are awash with code-mixed language, where speakers seamlessly blend multiple languages within a single utterance. This is particularly prevalent in informal settings like Twitter, where users often mix Hindi and English. While standard Natural Language Processing (NLP) tools are typically built for monolingual data and struggle with the inherent variability of code-mixed text, understanding the semantic roles of words (who did what to whom, where, when, why) is vital for many downstream NLP tasks like question answering and information retrieval. Existing SRL resources are primarily monolingual, leaving a significant gap in our ability to analyze this widespread form of communication. This paper tackles this challenge by creating a labeled dataset and a baseline system for SRL on Hindi-English code-mixed tweets.

Methods

The authors developed a comprehensive annotation scheme based on the Propbank (Proposition Bank) framework, which labels semantic roles associated with verbs. Their process involved:

  • Verb Frame Creation: They created verb frames for complex predicates (e.g., light verb constructions) that are common in code-mixed language, often requiring new frames not found in existing resources.
  • Dependency-to-Propbank Mapping: A key innovation is their mapping of Paninian dependency labels (used for Hindi) to Propbank labels. This leverages existing syntactic information to aid in semantic role annotation.
  • Rule-Based System: They propose a baseline rule-based system that uses the dependency tree structure and the established mappings to automatically identify and classify semantic roles. This system first identifies arguments and then classifies them using the learned mappings.
  • Handling Code-Mixing Nuances: The system also addresses specific challenges of code-mixed data, such as handling typos, colloquialisms, and pronoun dropping (where pronouns are omitted but inferable from context).

Datasets

The paper introduces a newly created dataset:

  • Hindi-English Code-Mixed Tweets Dataset: This dataset consists of 1460 tweets, totaling 20,949 tokens, manually labeled with Propbank semantic roles. This is the first dataset of its kind for Hindi-English code-mixed text.

Key Insights and Ideas

  • Code-Mixing is Rich and Complex: The study highlights that code-mixed tweets are not just simple word substitutions but involve complex predicate constructions and unique linguistic phenomena that require specialized handling.
  • Dependency Structures are Key: Leveraging Hindi dependency labels proved to be a powerful way to bridge the gap between syntax and semantics in code-mixed text, enabling more consistent annotation.
  • Pronoun Dropping is Common: The paper emphasizes the prevalence of pronoun dropping in Hindi-English code-mixed data, necessitating mechanisms to account for these "empty categories" in semantic role labeling.
  • Social Media Language is Unique: The authors had to create new frame files for many predicates due to the colloquial nature and novel usages found in social media, underscoring the limitations of traditional linguistic resources.

Key Results

  • Argument Identification Accuracy: The rule-based system achieved an impressive overall accuracy of 96.74% for identifying arguments.
  • Argument Classification Accuracy: The system was able to correctly classify 73.93% of the identified arguments with Propbank labels.
  • Distribution of Predicates: The dataset includes a significant number of unique complex predicates (462), with code-mixed complex predicates forming a substantial portion (232).

Limitations and Future Work

  • Rule-Based System Limitations: The authors acknowledge that their rule-based system, while effective, struggles with certain ambiguities, particularly between similar dependency labels that map to different Propbank roles (e.g., distinguishing between destination and location). This leads to lower accuracy for some specific labels.
  • Focus on Lexical Items: The current system primarily focuses on lexical items and does not yet incorporate symbols or emojis as potential arguments, which are common in social media.
  • Future Directions: The authors plan to expand the dataset and explore learning-based approaches for code-mixed SRL. They also aim to compare their system's performance with models trained on monolingual datasets.

Summary updated

BibTeX
@inproceedings{pal2019dataset,
  author = {Riya Pal and Dipti Sharma},
  title = {{A Dataset for Semantic Role Labelling of Hindi-English Code-Mixed Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure