TL;DR
This paper tackles the challenge of building robust NLP tools for Occitan, a Romance language with significant dialectal variation and limited resources. They extend an existing Occitan treebank to include three new dialects, creating a multi-dialect corpus. Their experiments show that models trained on Occitan dialects outperform larger, delexicalized models trained on other Romance languages, highlighting the importance of dialect-specific data.
What Problem Does This Paper Address?
Occitan, a Romance language spoken in southern France, presents a significant challenge for Natural Language Processing (NLP) due to its lack of standardization and rich dialectal variation. While previous efforts have created resources like lexicons and a treebank, these were primarily based on the Lengadocian dialect. This leaves a gap in developing NLP tools that can effectively process the full spectrum of Occitan varieties. The authors aim to bridge this gap by expanding the existing treebank to include three additional dialects (Gascon, Provençau, and Lemosin) and investigating the most effective methods for training robust parsing models that can generalize across these dialects.
Methods
The core of this work involves extending an existing Occitan treebank and conducting parsing experiments.
- Corpus Extension: The authors leveraged the RESTAURE corpus to incorporate texts from Gascon, Lemosin, and Provençau into their existing Lengadocian treebank. This involved lemmatizing and POS-tagging the new content, then converting it to Universal Dependencies (UD) tags.
- Syntactic Annotation: For syntactic annotation, they employed a strategy of pre-annotating samples using a delexicalized cross-lingual parsing model trained on French, Italian, and Portuguese UD corpora, followed by manual correction. This approach was chosen to expedite the annotation process.
- Parsing Experiments: The paper explores three main scenarios for training parsing models:
- Delexicalized cross-lingual parsing: Using a model trained on UD corpora of Italian, Portuguese, and French.
- Direct parsing transfer: Using a lexicalized model trained on Occitan (specifically Lengadocian).
- Delexicalized cross-dialectal parsing: Using a delexicalized model trained on Occitan dialects. They further refine scenarios 2 and 3 by investigating the impact of adding training material from other dialects to a Lengadocian-based model.
Datasets
- Initial Treebank (TTB): The paper builds upon the existing Tolosa Treebank (TTB), which initially contained 19K tokens of Lengadocian texts across five genres.
- Extended TTB: The extended corpus now comprises 25K tokens, incorporating approximately 4K tokens from Gascon, 1.3K from Lemosin, and 1.2K from Provençau. The genre diversity in the added dialects is noted as being less than in the Lengadocian portion.
- External UD Corpora: For delexicalized cross-lingual parsing, the authors utilized UD corpora from Italian, Portuguese, and French. These corpora are significantly larger than the Occitan data, with the French, Italian, and Portuguese combined corpus containing around 900K tokens.
Key Insights and Ideas
- Dialect-Specific Data Trumps Large Generic Data: The most striking finding is that even a relatively small amount of annotated data in Occitan dialects (around 20K tokens) leads to better parsing performance than a much larger (900K tokens) delexicalized model trained on other Romance languages. This strongly suggests that for under-resourced languages with significant variation, investing in target-language data is crucial.
- Delexicalization Benefits Occitan: Delexicalized models trained on Occitan dialects generally outperformed their lexicalized counterparts. This indicates that abstracting away from specific word forms helps models generalize better across dialectal lexical variations.
- Syntactic Variation Less Impactful Than Expected: Contrary to expectations, the syntactic variations observed, particularly in Gascon, did not seem to significantly hinder model performance. This suggests that while lexical differences are a major hurdle, syntactic differences might be more manageable for current parsing models.
- The Power of Small, Targeted Datasets: The paper underscores the value of even small, carefully curated datasets in a target language. The performance of models trained on Occitan data, even when much smaller than external resources, highlights this point.
Key Results
- Parsing Performance: The best-performing models for Lengadocian and Provençau were achieved by a delexicalized model trained on a combination of Lengadocian and Gascon. For Gascon and Lemosin, lexicalized models trained on Lengadocian plus the respective dialect performed best.
- Overall Best Model: The delexicalized model trained on Lengadocian and Gascon was identified as the most useful across the board, performing comparably well on Lemosin and achieving strong results on Gascon.
- Delexicalized vs. Lexicalized: Delexicalized models consistently outperformed lexicalized models trained on Occitan, by an average of 1-3 points in LAS.
- Cross-Lingual Model Performance: The delexicalized cross-lingual model trained on Italian, Portuguese, and French performed the worst, with a 10-15 point difference in LAS compared to the best Occitan-specific models.
Limitations and Future Work
- Small Sample Sizes: The authors acknowledge that the added samples for Gascon, Lemosin, and Provençau are still relatively small (around 900 tokens each), which can impact the robustness of machine learning models.
- Genre Diversity: The Lemosin and Provençau sections of the corpus are limited to literary texts, lacking the genre diversity found in the Lengadocian and Gascon portions.
- Future Work: The authors plan to repeat these evaluations once the samples are extended to confirm their findings. They also aim to continue corpus enrichment and develop more efficient parsing models to facilitate the creation of robust NLP tools for all Occitan texts. Further investigation into whether variety-based parsing is necessary in the future is also implied.
Summary updated