TL;DR
This paper introduces the DISRPT 2019 Shared Task, a novel initiative to standardize and advance automatic discourse parsing by focusing on elementary discourse units (EDUs) and connective detection across multiple languages and formalisms. The task's success lies in its creation of a unified dataset and evaluation framework, paving the way for more robust and generalizable discourse analysis tools.
What Problem Does This Paper Address?
Discourse parsing, the task of understanding the structure and relationships between parts of a text, has seen significant progress, but a major bottleneck remains: the accurate identification of the basic building blocks of discourse, known as Elementary Discourse Units (EDUs). Current research often relies on "gold standard" segmented EDUs, which simplifies the problem and doesn't reflect real-world scenarios where segmentation is also a challenge. This makes it difficult to compare system performance across different studies and limits our understanding of how well discourse parsers perform in realistic settings. Furthermore, the field lacks a standardized, multilingual benchmark for this crucial sub-task, hindering progress and cross-linguistic research.
Methods
The DISRPT 2019 Shared Task addressed this by organizing a competition focused on two core tasks:
- Elementary Discourse Unit (EDU) Segmentation: Identifying the minimal units of text that form the basis of discourse structures.
- Connective Detection: Identifying explicit words or phrases (connectives) that signal relationships between EDUs.
The task provided a unified dataset derived from 15 existing datasets across 10 languages, all converted to a common CoNLL-U format. This standardization was key to enabling fair comparison of systems. Systems were evaluated on both "treebanked" data (with gold standard annotations) and "plain tokenized" data (unannotated text), simulating different levels of input quality.
Datasets
The shared task utilized a curated collection of 15 datasets from 10 languages, covering various discourse formalisms like Rhetorical Structure Theory (RST) and Segmented Discourse Representation Theory (SDRT). These datasets range in size from a few thousand tokens to over a million. Notably, for some datasets, the original texts could not be publicly released due to licensing restrictions, and were instead represented by underscores in the public repository, requiring users to obtain the original texts separately. The data was provided in two formats:
- Treebanked data (*.conll): Includes gold standard EDU segmentation, sentence splits, POS tags, and dependency parses.
- Plain tokenized data (*.tok): Contains only tokenized text, simulating a more challenging scenario without pre-existing structural annotations.
Key Insights and Ideas
- The "Safe Strategy" Advantage: Systems often achieved higher precision than recall in EDU segmentation. This suggests that simple, "safe" strategies, like assuming sentence beginnings mark segment boundaries, are effective for precision but struggle with identifying more nuanced, non-obvious segmentation points.
- Dataset Size Matters: Performance, particularly for connective detection, generally correlates with dataset size. Larger datasets likely contain a more exhaustive inventory of connectives and their disambiguating contexts, leading to better generalization.
- Syntax is Not Central (for Connectives): The results for connective detection showed similar performance between treebanked and plain tokenized data, indicating that explicit syntactic information is not a primary driver of success for this task.
- Generic Models Can Excel: The top-performing system, ToNy, demonstrated remarkable success using a generic sequence labeling approach with contextualized embeddings, highlighting the power of modern NLP architectures even without extensive feature engineering.
- Multilingualism is Challenging: The variability in annotation guidelines and the performance differences across languages underscore the ongoing challenge of creating truly universal discourse parsing models.
Key Results
- EDU Segmentation: The ToNy system achieved the best macro-averaged F1 score (90.11) across datasets and performed best on the majority of individual datasets. Other strong performers included GumDrop (88.11 F1), IXA (87.18 F1), and DFKI RF (84.56 F1).
- Connective Detection: ToNy also led in connective detection on plain tokenized data, while GumDrop showed a significant lead on gold syntax data for the Turkish dataset.
- State-of-the-Art Improvement: For the established RST-DT benchmark, two systems (ToNy and GumDrop) surpassed previous state-of-the-art scores for EDU segmentation.
Limitations and Future Work
- Realistic Segmentation: The paper acknowledges that even the "plain tokenized" scenario is still easier than true end-to-end parsing, as it assumes correct tokenization. Future work could explore systems that handle tokenization errors.
- Low-Resource Languages: While the task included multiple languages, performance on less-studied languages still lags significantly behind well-resourced ones. Further research is needed to improve robustness in low-resource settings.
- Dataset Maintenance and Expansion: The authors plan to maintain the DISRPT dataset and potentially expand it for future iterations of the workshop, suggesting a continued effort to foster research in this area.
- System Architectures: The success of ToNy's generic approach suggests that further exploration of end-to-end neural models for discourse parsing is warranted, potentially leading to more unified and powerful systems.
Summary updated