A Discourse Signal Annotation System for RST Trees

Luke Gessler, Yang Liu, Amir Zeldes

TL;DR

This paper introduces a novel web-based system that allows for the simultaneous annotation of Rhetorical Structure Theory (RST) trees and the specific linguistic signals that indicate the relationships between discourse units. This bridges a critical gap in existing RST annotation tools, enabling a more comprehensive and integrated analysis of text structure and its signaling mechanisms.

What Problem Does This Paper Address?

Analyzing how language users signal relationships between clauses and sentences (discourse relations) is crucial for understanding text coherence. Rhetorical Structure Theory (RST) provides a powerful framework for modeling these relationships as hierarchical trees. However, existing tools for RST annotation have struggled to effectively link these hierarchical structures with the explicit linguistic signals (like conjunctions, adverbs, or even syntactic patterns) that indicate these relations. This has made it difficult to conduct empirical studies that simultaneously examine both the structure and the signaling of discourse. The paper addresses this by developing an annotation system that integrates RST tree annotation with the identification and annotation of discourse signals within a single, unified interface.

Methods

The core contribution is the development of a signal annotation system built on top of an existing web-based RST annotation tool called rstWeb. This new system allows annotators to:

  • View and Edit Signals: A dedicated sidebar, accessible from a button next to each relation in the RST tree, allows for the annotation of discourse signals.
  • Link Signals to Relations: Signals can be directly associated with specific RST relations.
  • Annotate Signal Tokens: Signals can be linked to specific tokens (words or phrases) within the text that constitute the signal. This is a significant advancement over previous methods that often lacked precise token-level anchoring.
  • Flexible Signal Categorization: The system supports configurable signal types and subtypes, allowing for a rich and adaptable annotation scheme.
  • Integrated Workflow: The system avoids separate modes for tree and signal annotation, allowing annotators to work on both simultaneously within the same environment, streamlining the process.
  • Automatic Tokenization: For alphabetic languages, the system incorporates automatic tokenization using a Python port of the TreeTagger tokenizer, which is essential for anchoring signals to specific words.

Datasets

The paper discusses the use of existing corpora and introduces a new benchmark for evaluation:

  • RST Signalling Corpus (RST-SC): The system was developed and evaluated using data from the RST-SC, which already contains RST trees and some signal annotations.
  • GUM Corpus: The authors are currently using their interface to annotate RST signals in the GUM corpus, a large, multi-genre, richly annotated corpus.
  • Pilot Data Set: For evaluation, a pilot data set of three documents from RST-SC, containing 506 tokens and over 90 signals, was re-annotated.

Key Insights and Ideas

  • Unified Annotation is Crucial: The most significant insight is that a truly integrated approach to annotating both RST trees and their associated signals is essential for advancing empirical research in discourse analysis. Previous tools forced a separation, hindering comprehensive study.
  • Token-Level Anchoring Matters: Precisely linking signals to specific words or phrases in the text is vital for accurate and reproducible analysis, especially when comparing human annotations to computational models.
  • Flexibility in Signal Types: The ability to define and use a wide range of signal types and subtypes is necessary to capture the diverse ways coherence is signaled across different genres and text types.
  • User Experience Enhances Annotation Quality: The new interface was reported to be significantly easier and faster to use, suggesting that improved tools can lead to better annotation quality and inter-annotator agreement.

Key Results

  • Improved Annotation Accuracy: In a re-annotation task on a pilot data set, the new system achieved higher agreement scores (Cohen's kappa of 0.77) compared to previous work (0.52) for anchoring RST-SC signals to specific tokens.
  • Increased Efficiency: Annotators reported the new interface was "much easier and faster to work with."
  • Enhanced Suitability: Feedback from experienced annotators indicated the interface is "much more suited to the signal annotation task."

Limitations and Future Work

  • Tokenization for Non-Alphabetic Languages: The current automatic tokenization is limited to alphabetic languages. The authors acknowledge that built-in tokenization for Asian languages and morphologically rich languages is still needed, though pre-tokenized data can be imported.
  • Extending Signal Schemes: The signal type and subtype schemes, currently based on the Wall Street Journal corpus, need to be extended to cover the broader range of genres present in corpora like GUM.
  • Search and Visualization: The authors are working on developing search and visualization facilities (e.g., using the ANNIS platform) to explore the annotated data more effectively, including interactive ways to explore signaling tokens and their relation to RST trees.
  • Discovering New Signal Types: A key goal is to learn which new types of signals are needed for different text types and genres, which will inform the development of more general models for discourse relation identification.
  • Comparison with Neural Models: The authors plan to compare human-annotated signal spans with words most attended to by neural models for automatic relation classification.

Summary updated

Abstract

This paper presents a new system for open-ended discourse relation signal annotation in the framework of Rhetorical Structure Theory (RST), implemented on top of an online tool for RST annotation. We discuss existing projects annotating textual signals of discourse relations, which have so far not allowed simultaneously structuring and annotating words signaling hierarchical discourse trees, and demonstrate the design and applications of our interface by extending existing RST annotations in the freely available GUM corpus.
BibTeX
@inproceedings{gessler2019discourse,
  author = {Luke Gessler and Yang Liu and Amir Zeldes},
  title = {{A Discourse Signal Annotation System for RST Trees}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure