TL;DR
This paper introduces RST-Tace, a novel tool for automatically comparing and evaluating Rhetorical Structure Theory (RST) trees. It implements a modified version of Iruskieta's method, allowing for robust comparison of discourse structures even when lower-level annotations differ, and addresses the challenge of measuring inter-annotator agreement in this complex domain.
What Problem Does This Paper Address?
Rhetorical Structure Theory (RST) is a widely used framework for analyzing text organization by identifying the relationships between elementary discourse units (EDUs). However, the inherent subjectivity in discourse analysis means that different annotators can produce valid but distinct RST tree structures for the same text. Distinguishing genuine annotation differences from errors is crucial for ensuring the reliability of RST analyses. While methods exist for comparing RST trees, they often struggle with variations in lower-level annotations (like constituent attachment points) significantly impacting overall agreement scores. Furthermore, manually comparing and evaluating these trees is a time-consuming and resource-intensive task. This paper addresses the need for an automated, robust, and efficient tool that can accurately compare and evaluate RST trees, even when faced with annotation discrepancies.
Methods
The core of RST-Tace is its implementation of a modified Iruskieta's method for comparing RST trees. This method focuses on four key factors: Constituent (C), Attachment point (A), Nuclearity (N), and Relation (R). Crucially, it allows for comparison even when constituents don't perfectly align, prioritizing the "central subconstituent" (CS) as the primary matching criterion.
Key methodological aspects include:
- Harmonized Segmentation: The tool assumes discourse segmentation (identifying EDUs) is already harmonized between annotators.
- Central Subconstituent (CS) Matching: The primary condition for comparing relations is that their central subconstituents must be identical. However, the method accommodates cases with multinuclear relations where multiple CSs exist, requiring at least one identical CS for comparison.
- Cost Matrix and Hungarian Algorithm: To handle the ambiguity of matching relations between two trees, RST-Tace constructs a cost matrix based on different degrees of equivalence (e.g., identical CS, matching attachment points, or at least one identical CS). The Hungarian algorithm is then employed to find the optimal association of relations between the two trees, minimizing the overall cost.
- Automated Evaluation: Once relations are optimally associated, the tool automatically compares them based on the four factors (C, A, N, R) and calculates F-measure and inter-annotator agreement scores.
Datasets
The paper mentions using the RST German Learner Treebank for evaluation. Specifically, examples are drawn from texts like CMN_008 and DEU_006. The authors also refer to a "self-built RST treebank" for presenting an example of comparison. No new datasets were introduced.
Key Insights and Ideas
- Focus on Robust Comparison: The most significant insight is the development of a method that doesn't require perfect alignment of all annotation elements (like constituents) to deem two relations comparable. This acknowledges the reality of inter-annotator variability.
- Central Subconstituent as a Key Anchor: Identifying and using the "central subconstituent" as the primary matching criterion is a clever way to anchor comparisons to the most important part of a satellite span.
- Hungarian Algorithm for Optimal Association: Applying the Hungarian algorithm to find the optimal matching between relations in two trees is a sophisticated approach to resolve the ambiguity inherent in comparing potentially different structures.
- Automated Evaluation for Scalability: The development of an automated tool like RST-Tace is crucial for enabling large-scale comparison and evaluation of RST annotations, which is otherwise prohibitively time-consuming.
Key Results
The paper demonstrates the functionality of RST-Tace through an example comparison of two RST trees from the DEU_006 text in the RST German Learner Treebank. Figure 6 presents a detailed comparison table and the evaluation results.
- Quantitative Results: For the DEU_006 example, the F-measure for comparing RST trees was reported as 0.737. Inter-annotator agreement scores for Nuclearity, Relation, Constituent, and Attachment point were 0.71, 0.698, 0.57, and 0.59, respectively. The overall F-measure for the comparison was 0.804 (33 out of 40 relations matched).
- Qualitative Findings: The comparison table (Figure 6) shows various matching outcomes, including "Completely identical CS," "C1=C2 and A1=A2," "C1=A2 and A1=C2," and "Partially identical CS." This highlights the tool's ability to categorize different levels of agreement.
Limitations and Future Work
- Limitations:
- The tool assumes harmonized discourse segmentation (EDUs are the same across annotators). Differences in segmentation would require pre-processing.
- The current implementation's statistical evaluation part is under active development.
- The paper focuses on agreement rather than disagreement, which might be a limitation for certain research questions.
- Future Work:
- Development of a user-friendly interface to make the tool more accessible.
- Implementation of more sophisticated statistical analysis for larger datasets and RST treebanks.
- Potentially extending the tool to handle disagreement analysis more comprehensively.
Summary updated