TL;DR
This paper explores the challenging task of classifying discourse relations (how sentences and clauses connect logically) across different languages and theoretical frameworks, specifically focusing on "zero-shot" learning where models must generalize to unseen data. They find that while challenging, multilingual models can achieve surprisingly good performance, with label set similarity being a more critical factor than language family or framework.
What Problem Does This Paper Address?
Classifying discourse relations is crucial for understanding text coherence and is vital for downstream NLP tasks like summarization and machine translation. However, this field faces significant hurdles:
- Data Scarcity: Annotated discourse data is rare, especially for languages other than English.
- Theoretical Divergence: Different linguistic frameworks define discourse units and relation types differently, making cross-framework transfer difficult.
- Limited Transfer: Existing work on cross-lingual transfer for discourse relations is limited, hindering the development of robust multilingual systems.
This paper aims to bridge this gap by investigating zero-shot learning for multilingual discourse relation classification, exploring how models trained on one set of languages/frameworks can perform on unseen ones, and identifying factors that facilitate or hinder this transfer.
Methods
The authors leverage multilingual transformer-based models, specifically mBERT, DistilmBERT, and XLM-ROBERTa, which are pre-trained on vast amounts of text in multiple languages. Their approach involves:
- Joint Training: Training a single multilingual model on a combination of available discourse corpora.
- Label Filtering: A post-processing step to ensure predicted labels belong to the target framework, preventing the model from hallucinating labels not present in the target data.
- Feature Augmentation: Injecting special tokens at the beginning of the input sequence to inform the model about the language, corpus, and framework of the data. This helps the model contextualize its predictions.
- Zero-Shot Evaluation: Testing the trained multilingual models on datasets from languages or frameworks they have not seen during training. They explore zero-shot transfer based on language families, frameworks, and label set similarity.
- Jaccard Similarity: Used to quantify the overlap between label sets of different corpora, serving as a metric to predict transferability.
Datasets
The study utilizes datasets from the DISRPT 2023 Shared Task, which provides a unified format for discourse annotation across multiple languages and theoretical frameworks. Specifically, they use 26 datasets covering 13 languages and 4 theoretical frameworks:
- PDTB (Penn Discourse Treebank)
- RST (Rhetorical Structure Theory)
- DEP (Dependency structures)
- SDRT (Segmented Discourse Representation Theory)
Key Insights and Ideas
- Label Similarity is King: The most significant factor for successful zero-shot transfer is the similarity between the label sets of the source and target corpora, often more so than language family or framework.
- Multilingual Models are Powerful: Even base-sized multilingual models, when augmented with feature engineering (like language/framework tokens) and label filtering, can perform comparably to or even outperform monolingual models on unseen data.
- Zero-Shot is Challenging but Promising: While accuracy drops significantly in zero-shot settings, the results are not always catastrophic, especially when label sets overlap. This offers a hopeful outlook for including under-represented languages.
- Frameworks Matter, But Less Than Labels: While different frameworks introduce complexities, the overlap in relation types (label similarity) is a stronger predictor of transfer success.
- Feature Augmentation Helps: Informing the model about the data's origin (language, corpus, framework) through special tokens significantly improves performance.
Key Results
- Multilingual vs. Monolingual: Multilingual models generally perform better than monolingual ones, especially on smaller datasets, with improvements ranging from moderate to significant.
- Feature Augmentation Gains: Models with feature augmentation consistently outperform baseline models without it, with the combination of language, corpus, and framework tokens being particularly effective.
- Zero-Shot Performance: Accuracy drops in zero-shot settings, but some English corpora achieved near-zero accuracy, as expected when their labels were entirely absent from training. However, for some OOD (Out-Of-Domain) datasets, performance even improved when trained on a broader set of similar corpora.
- HITS System Comparison: The authors' multilingual models, even with base-sized transformers, achieved performance close to the state-of-the-art HITS system from DISRPT 2023, often within a 1-3% margin, despite HITS using larger models and more complex optimizations.
Limitations and Future Work
- Limited Language Coverage: The study is constrained by the availability of annotated discourse corpora, limiting evaluation to a dozen languages and not fully representing the diversity of language families.
- Annotation Disparity: Differences in annotation detail and overlap between frameworks and even within the same framework can affect transferability. Ideally, label sets would be more aligned for precise evaluation.
- Future Work: The authors suggest that their findings are a hopeful step towards including under-represented languages in discourse analysis and integrating discourse analysis into other NLP tasks. Further research could explore more sophisticated methods for handling label set divergence and expanding language coverage.
Summary updated