Abstract
Recent claims of massive breakthroughs in Rhetorical Structure Theory (RST) discourse parsing, often citing error reductions of over 50%, have been largely attributed to the adoption of neural networks and distributed representations. However, this replication study reveals that much of this perceived progress is an artifact of inconsistent evaluation metrics. By replicating nine recent parsers and standardizing evaluation using micro-averaging, the authors found that actual progress is a more modest 16% error reduction. This paper advocates for the adoption of standard syntactic Parseval metrics to ensure transparent and comparable results in future discourse parsing research.
Introduction
Discourse parsing is a fundamental task in natural language processing that involves uncovering the hierarchical rhetorical structure of a text. By identifying relations such as "Explanation" or "Contrast" between text segments, parsers build a tree-like representation of a document's discourse. Between 2013 and 2017, the field saw a flurry of publications claiming significant performance leaps on the standard RST Discourse Treebank (RST-DT), with some reporting relative error reductions as high as 51%. These gains were primarily credited to modern neural architectures and word embeddings.
The Problem: Inconsistent Evaluation
The authors of this study suspected that these dramatic improvements might be less about algorithmic innovation and more about evaluation methodology. Specifically, they identified a lack of transparency in how different research groups were calculating their scores. The core issue lies in the choice between micro-averaging and macro-averaging. While traditional baselines used micro-averaging (computing a global score across all spans in all documents), many recent neural parsers shifted to macro-averaging (averaging per-document scores), which consistently yields higher numbers on the RST-DT dataset. This shift created an illusion of rapid progress.
Methods and Standardization
To resolve this, the authors conducted a comprehensive meta-evaluation by collecting raw predictions from the authors of eight major studies and reimplementing the standard RST-Parseval procedure.
As shown in the figure above, the study also explored structural transformations. They argued that the traditional RST-Parseval metric is "too easy" because it counts redundant nodes, such as the root and leaves, which artificially inflates agreement. To provide a more rigorous assessment, they proposed using standard Parseval metrics applied to labeled attachment decisions on binary trees, a method common in syntactic parsing.
Key Results
When evaluated on a level playing field using micro-averaging, the "massive" gains of recent years largely vanished. The purported 51% relative error reduction on fully-labeled structures dropped to just 16%.
The sensitivity of these models to the choice of averaging is stark. For instance, the JE14 DPLP parser shows a jump in performance when moving from micro-averaged results (seen in the table above) to macro-averaged metrics. This ~4 point difference was the primary driver of the "purported progress" reported in recent literature. Furthermore, when using the more stringent "Standard Parseval" metric, the best system achieved an F1 score of only 46.3, significantly below the human agreement level of 55.0, highlighting how much room for improvement remains.
Conclusion
This replication study highlights an urgent need for standardization in the discourse parsing community. The perceived "neural revolution" in RST parsing was partly an illusion created by a shift in evaluation metrics. While neural models do provide genuine improvements, the progress is more incremental than previously thought. The authors have released their evaluation tools to help the community move toward more transparent and comparable research practices.
Summary updated