TL;DR
This paper introduces a novel system for annotating "discourse signals" – words and phrases that indicate relationships between parts of a text – within the Rhetorical Structure Theory (RST) framework. By integrating signal annotation directly into an existing RST treebuilding tool, they enable simultaneous annotation of both the hierarchical text structure and the explicit linguistic cues that signal these relationships, a capability previously lacking.
What Problem Does This Paper Address?
Understanding how language connects ideas is crucial for analyzing text. Rhetorical Structure Theory (RST) provides a powerful framework for modeling these connections as hierarchical trees, but existing tools have struggled to simultaneously annotate both the tree structure and the specific words or phrases (discourse signals) that signal these relationships. While some approaches annotate signals separately or in less structured ways, there's been a gap in tools that can link explicit signals directly to their corresponding nodes and relations within an RST tree. This makes it difficult to empirically study how writers use signals and how readers interpret them in relation to the overall text structure.
Methods
The core contribution is the development of a signal annotation system built on top of an existing web-based RST annotation tool called rstWeb. This new system allows annotators to:
- Simultaneously Annotate: Link discourse signals directly to specific relations within an RST tree.
- Flexible Signal Definition: Define custom signal types and subtypes (e.g., "discourse marker," "semantic," "lexical chain") and associate them with any number of text tokens.
- Token-Level Anchoring: Precisely link signals to specific words or phrases in the text, a significant improvement over previous methods that often lacked this granularity.
- Integrated Workflow: Avoids switching between separate tools, allowing annotators to build RST trees and annotate signals within a single interface.
- Data Model Extension: The
.rs3XML format used for RST data was extended to.rs4to accommodate signal information, ensuring backward compatibility. - Automatic Tokenization: The system incorporates automatic tokenization for alphabetic languages using a Python port of the TreeTagger tokenizer, a necessary step for anchoring signals to specific words.
Datasets
- RST Signalling Corpus (RST-SC): The system was evaluated by re-annotating a pilot dataset from RST-SC, containing 506 tokens and over 90 signals. This corpus is based on existing RST trees and adds explicit signal annotations.
- GUM Corpus: The authors are currently using their interface to annotate RST signals in the GUM corpus, a large, richly annotated corpus spanning multiple genres. This allows for extending signal annotation schemes beyond the original Wall Street Journal focus of RST-SC.
Key Insights and Ideas
- Bridging the Gap: The most significant insight is that by integrating signal annotation directly into RST treebuilding, researchers can now study the interplay between explicit linguistic cues and hierarchical text structure in a unified way.
- Beyond Explicit Connectives: The system supports annotating a broader range of signals than just traditional discourse markers, including semantic and syntactic features, which are often overlooked.
- Flexibility is Key: The ability to define custom signal types and associate them with any token span is crucial for adapting to diverse text genres and annotation goals.
- Tokenization is Essential: The paper highlights that for effective signal annotation linked to text, robust tokenization is a prerequisite, especially for languages with complex morphology.
Key Results
- Improved Annotation Quality: In a pilot study, re-annotating RST-SC data with the new system resulted in higher agreement (90.9% identical, kappa 0.77) compared to a previous study (86.0% identical, kappa 0.52) that used separate tools.
- Enhanced User Experience: Annotators reported the new interface was significantly easier and faster to use than previous methods.
- Broader Applicability: The system is being used to annotate the GUM corpus, demonstrating its utility for multi-genre text analysis and the extension of signal annotation schemes.
Limitations and Future Work
- Tokenization for Non-Alphabetic Languages: The current system's automatic tokenization is limited to alphabetic languages. Future work includes developing or integrating tokenization for Asian and morphologically rich languages.
- Search and Visualization: The authors plan to develop advanced search and visualization tools (e.g., using the ANNIS platform) to explore the annotated data, enabling interactive analysis of signaling tokens and their relation to discourse trees.
- Genre-Specific Signals: Further research is needed to identify and characterize new types of signals relevant to different text genres, moving beyond the initial focus on news articles.
- Neural Model Comparison: The authors are exploring how human-annotated signal spans compare with words identified by neural models for automatic relation classification.
Summary updated