Annotating Shallow Discourse Relations in Twitter Conversations

Tatjana Scheffler, Berfin Aktaş, Debopam Das, Manfred StedeNAACL 2019

TL;DR

This paper introduces a novel dataset and methodology for annotating discourse relations in Twitter conversations, adapting the established Penn Discourse Treebank (PDTB) style. The study reveals that Twitter conversations exhibit distinct discourse patterns compared to formal written text, with a higher prevalence of contingency relations and more fragmented argument structures, paving the way for deeper analysis of social media communication.

What Problem Does This Paper Address?

Understanding how people connect ideas and build arguments in text is crucial for many NLP tasks, from summarization to dialogue systems. The Penn Discourse Treebank (PDTB) has been a foundational resource for this, providing annotations of discourse relations (like cause-effect, contrast, elaboration) in formal written text. However, social media, particularly Twitter, presents a unique communication environment. Its brevity, rapid-fire nature, and informal style likely lead to different ways of expressing these relations. Existing discourse annotation frameworks, developed for more formal texts, may not adequately capture the nuances of Twitter conversations. This paper addresses the gap by applying a PDTB-style annotation to Twitter, aiming to investigate and characterize these genre-specific differences in discourse structure.

Methods

The authors adapted the PDTB's "shallow" annotation approach, focusing on explicit discourse connectives (words like "and," "but," "because") and their immediate arguments within a single tweet. They started with a list of 100 explicit connectives from the PDTB and supplemented it with a few new ones identified in their Twitter data. The annotation process involved:

  1. Connective Identification: Identifying explicit discourse connectives.
  2. Argument Span Identification: Pinpointing the text spans (arguments) that the connective relates.
  3. Sense Labeling: Assigning a discourse relation sense (e.g., CONTINGENCY, EXPANSION) based on the PDTB's hierarchical taxonomy.

A key methodological challenge was adapting argument span selection criteria to accommodate the fragmented and often elliptical nature of Twitter text, where arguments might be noun phrases or short clauses rather than full sentences. They also developed specific guidelines to handle social media specific elements like hashtags and emojis. To ensure reliability, they conducted an inter-annotator agreement (IAA) study on a subset of their data.

Datasets

The paper introduces a newly annotated corpus of 185 Twitter threads, comprising 1756 tweets. This corpus was collected from the Twitter stream over several days in December 2017 and January 2018, without filtering for specific topics. For the initial phase of annotation, they focused on intra-tweet relations, meaning both the connective and its arguments were contained within the same tweet. This dataset is a novel benchmark for studying discourse relations in social media conversations, distinct from existing corpora like the PDTB which focus on formal written text.

Key Insights and Ideas

  • Twitter's Discourse is Different: Explicit discourse relations are frequent in Twitter conversations, but their distribution and expression differ significantly from formal written text.
  • Contingency Reigns Supreme: Twitter conversations show a marked increase in CONTINGENCY relations (especially causal and conditional) compared to the PDTB. This suggests a greater emphasis on explaining "why" or "under what conditions" in social media interactions.
  • Fragmented Arguments are the Norm: Arguments of connectives in Twitter are often not full syntactic clauses but rather elliptical phrases, noun phrases, or even single words. This reflects the platform's brevity and informal style.
  • A Few Connectives Do the Heavy Lifting: A small set of common connectives (like "and," "but," "if") account for the majority of explicit relations, similar to spoken language patterns.
  • Challenges in Annotation: The informal nature of Twitter, including abbreviations, varied spellings, and the ambiguity of what constitutes an argument span (especially with hashtags and emojis), poses significant challenges for annotation consistency.

Key Results

  • High Frequency of Relations: Over 40% of tweets in their corpus contained at least one tweet-internal explicit discourse relation.
  • Top Connectives: The most frequent connectives in their Twitter corpus were "and" (30.0%), "but" (16.2%), and "if" (7.3%), with "and" and "but" also being prominent in the PDTB but with different relative frequencies.
  • Sense Distribution Shift: CONTINGENCY relations constituted 28.0% of senses in their Twitter corpus, compared to 18.7% in the PDTB. Conversely, COMPARISON and TEMPORAL relations were less frequent on Twitter (24.3% and 14.3% respectively) than in the PDTB (28.8% and 19.0%).
  • Inter-Annotator Agreement: IAA was generally good, with 70% agreement for connective detection and 89% for Arg2 span selection. However, exact Arg1 span selection showed lower agreement (62%), attributed to the difficulty in identifying utterance and clause breaks in Twitter. Sense agreement also showed good results, with Level-1 sense agreement at 88%.

Limitations and Future Work

The authors acknowledge that their current study is a pilot study and only covers intra-tweet relations. This means they haven't yet annotated relations where the connective and its arguments span across multiple tweets, nor have they addressed implicit discourse relations.

Future work will focus on:

  • Annotating Inter-Tweet Relations: Extending the annotation to cover relations that bridge multiple tweets.
  • Annotating Non-Explicit Relations: Incorporating implicit discourse relations, which are common in conversations.
  • Refining Annotation Guidelines: Further specifying guidelines to handle the unique phenomena of social media text (elliptical constituents, hashtags, emoji, abbreviations, etc.) and improving consistency.
  • Developing a Shallow Discourse Parser: Using the annotated corpus to build a parser specifically for English social media text.

Summary updated

BibTeX
@inproceedings{scheffler2019annotating,
  author = {Tatjana Scheffler and Berfin Aktaş and Debopam Das and Manfred Stede},
  title = {{Annotating Shallow Discourse Relations in Twitter Conversations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure