TL;DR
This paper tackles the challenge of automatically segmenting text into meaningful units for discourse analysis across various languages and genres. The authors propose a unified approach using machine learning (Random Forest and LSTM) and find that genre, rather than language, is a more significant factor in segmentation performance.
What Problem Does This Paper Address?
Discourse analysis, the study of how sentences and larger text units relate to each other to form coherent meaning, often relies on breaking down text into minimal "discourse units." These units are the building blocks for identifying relationships like cause-effect, contrast, or elaboration. However, different theoretical frameworks (like RST, PDTB, and SDRT) have varying definitions and methods for identifying these units, making it difficult to develop universal segmentation tools. This paper aims to bridge this gap by investigating a unified, theory-neutral approach to discourse unit segmentation that works across multiple languages and genres, a crucial first step for more complex discourse parsing.
Methods
The authors implemented two machine learning approaches:
- Random Forest: This method leverages a rich set of features derived from dependency trees (like grammatical function, part-of-speech tags of parent and current words, and word position) and, where available, constituency syntax features. They also incorporate features inspired by previous work on connective disambiguation.
- LSTM (Long Short-Term Memory): This recurrent neural network approach uses a smaller feature set, focusing on parent distance, grammatical function, POS tags, and word capitalization. They experimented with both pre-trained word embeddings and embeddings trained on the corpus itself, finding that pre-trained embeddings performed better for most datasets.
Both models were trained and evaluated on a collection of datasets from the DISRPT shared task, aiming for a unified segmentation strategy. They also established a simple baseline that segmented text at sentence boundaries or sentence-and-comma boundaries to gauge the effectiveness of their ML models.
Datasets
The paper utilizes data from the 2019 DISRPT (Discourse Relation Parsing and Treebanking) workshop, which includes 14 datasets from 9 languages. These datasets are annotated according to three main formalisms: RST (Rhetorical Structure Theory), PDTB (Penn Discourse Treebank), and SDRT (Segmented Discourse Representation Theory). The languages covered are Basque, Chinese, Dutch, English, French, German, Russian, Spanish, and Brazilian Portuguese. The paper notes that the Chinese Discourse Treebank and a Turkish dataset were not accessible for their experiments.
Key Insights and Ideas
- Genre Trumps Language: The most striking finding is that genre appears to be a much stronger predictor of segmentation performance than language. This suggests that models trained on one genre might generalize better to other languages within that same genre than to different genres within the same language.
- Unified Approach Feasibility: The study demonstrates that a single, theory-neutral segmentation approach is feasible, although performance varies significantly. This is a crucial step towards building more interoperable discourse processing tools.
- Feature Importance: For the Random Forest model, grammatical function, parent's part-of-speech tag, sentence position, and previous word's part-of-speech tag were consistently important features across datasets.
- PDTB's Unique Nature: The PDTB dataset, with its focus on identifying discourse connectives rather than strictly defined EDUs, presented a unique challenge. The authors found that their standard segmentation approaches performed poorly on PDTB, highlighting the need for specialized methods or a redefinition of "minimal units" for this type of annotation.
Key Results
- Both the RandomForest and LSTM models consistently outperformed the baseline across all datasets.
- The f-scores ranged from 59.60 (for Russian RRST with the baseline) to 94.47 (for English STAC with RandomForest).
- RandomForest achieved the best performance on 9 datasets, while LSTM was best on 4. The differences were often small.
- The LSTM approach showed a tendency to perform better on larger corpora, with the two models' performance converging.
- The baseline performance varied dramatically across datasets, from a high of 94.13 (English STAC) to a low of 59.60 (Russian RRST), underscoring the impact of genre and language on segmentation difficulty.
Limitations and Future Work
- Statistical Significance: The authors did not perform statistical significance tests, meaning conclusions drawn from small performance differences between models might not be robust.
- PDTB Segmentation: The paper acknowledges that their approach is not well-suited for the PDTB's specific definition of segmentation, which focuses on connectives. Further research is needed to unify segmentation across these different annotation styles.
- Genre-Based Grouping: The authors suggest that future work should focus on grouping datasets by genre rather than language to improve cross-lingual and cross-genre generalization. This could involve using multilingual word embeddings to create shared representations.
- Unifying Minimal Segments: The paper calls for a more refined, unified definition of a minimal discourse segment to facilitate cross-theory segmentation and improve the transferability of models.
Summary updated