TL;DR
This paper introduces EusDisParser, the first discourse parser for Basque, a low-resource language. By leveraging cross-lingual data and multilingual word embeddings, they demonstrate improvements over previous approaches, achieving a 34.78% F1 score for full discourse structure. This work highlights the potential of cross-lingual transfer learning for discourse parsing in under-resourced languages.
What Problem Does This Paper Address?
Discourse parsing, the task of analyzing the relational structure of text, is crucial for many natural language processing tasks like summarization and question answering. However, most existing discourse parsers are developed for high-resource languages like English, assuming large annotated datasets. For low-resource languages like Basque, where annotated data is scarce, building effective discourse parsers is a significant challenge. This paper aims to address this by developing the first discourse parser for Basque and investigating how to improve its performance by utilizing data from other languages.
Methods
The authors build upon a state-of-the-art transition-based constituent parser previously used for cross-lingual discourse parsing. Their approach involves:
- Monolingual System: Training a parser solely on the limited Basque data.
- Cross-lingual Systems:
- 'Src Only': Training a model on data from a source language (English, Spanish, or Portuguese) and evaluating it on the Basque test set. Hyperparameters are tuned using the Basque development set.
- 'Src+Tgt': Training a model on a combination of source language training data and the Basque development set. This strategy aims to see if including even a small amount of target language data improves performance.
- Multilingual Word Embeddings: A key innovation is the use of pre-trained multilingual word embeddings (specifically, FastText embeddings mapped using VecMap) to represent words from different languages in a shared vector space. This allows the model to learn cross-lingual representations.
- Feature Engineering: The parser utilizes a rich set of features, including words, POS tags, dependency information, and features related to the position and length of Elementary Discourse Units (EDUs).
Datasets
The paper utilizes several datasets for training and evaluation:
- Basque RST DT: A newly developed dataset for Basque, containing 88 abstracts from specialized domains and opinionative texts, annotated with 31 relations. This is the primary dataset for the Basque parser.
- English RST DT: A larger dataset with 385 documents from the Wall Street Journal, annotated with 56 relations.
- Spanish RST DT: A dataset with 267 texts annotated with 29 relations.
- Portuguese Corpora: A merged dataset of four existing corpora (CST-News, Summit, Rhetalho, CorpusTCC) containing 329 documents.
The Basque dataset was split into 38 documents for the test set and the remainder for the development set, following a split from a previous study.
Key Insights and Ideas
- Cross-lingual Transfer is Promising, but Nuanced: While cross-lingual approaches can improve performance, simply pooling data from multiple languages isn't always sufficient. The effectiveness depends on factors like language similarity and the quality of cross-lingual representations.
- Multilingual Embeddings are Crucial: The use of pre-trained multilingual word embeddings is essential for enabling effective cross-lingual transfer, allowing words from different languages to be represented in a common semantic space.
- Target Data is Key: Including even a small amount of target language data (Basque in this case) in the training set ('Src+Tgt' strategy) significantly boosts performance compared to relying solely on source language data ('Src Only'). This suggests that some language-specific information is vital.
- Data Sparsity is a Major Hurdle: The limited size of the Basque dataset is a significant constraint. The authors explicitly state that more annotated data is necessary for substantial improvements.
- Human vs. Machine Annotation Differences: Qualitative analysis reveals that while the parser excels at general relations like ELABORATION, humans tend to agree more on less common or more informative relations. The parser also uses a fixed set of relations, whereas humans can identify more nuanced ones.
Key Results
- Monolingual Basque Parser: The paper presents the first discourse parser for Basque, achieving a macro-F1 score of 34.78% for the full discourse structure using FastText embeddings. This is a significant baseline for future work.
- Cross-lingual Performance:
- The 'Src Only' strategy yielded lower results than the monolingual system, with a best macro-F1 of 22.14%. This suggests that without target language data, cross-lingual transfer is less effective.
- The 'Src+Tgt' strategy showed improvement, achieving a macro-F1 of 33.88%, which is close to the monolingual performance and significantly better than 'Src Only'.
- Embeddings Impact: Using FastText embeddings generally outperformed embeddings trained on the smaller Elhuyar corpus for Basque.
- Comparison to Previous Work: The authors' monolingual systems for English and Portuguese showed slight improvements over previous cross-lingual results reported by Braud et al. (2017), attributed to better word embeddings.
Limitations and Future Work
- Limited Data: The most significant limitation is the small size of the Basque annotated corpus. The authors explicitly state that more data is needed to improve results.
- CU Detection: The parser's central unit (CU) detection is less reliable than dedicated CU detectors, potentially due to its left-to-right or bottom-up annotation style.
- Relation Set: The parser uses a fixed set of rhetorical relations, which may not capture the full complexity of human annotation.
- Future Work:
- More Annotation: Continue annotating more Basque data to improve parser performance.
- Improved Cross-lingual Representations: Explore better multilingual word embeddings or other cross-lingual transfer techniques.
- Top-Down CU Detection: Investigate top-down approaches for central unit detection.
- Fine-grained Relations: Develop a system that can identify more informative and less common relations, addressing class imbalance.
- Pre-annotation: Leverage the parser to provide semi-automatic pre-annotations, speeding up the manual annotation process.
Summary updated