Modeling Orthographic Variation in Occitan’s Dialects

Zachary William Hopton, Noëmi AepliNAACL 2024

TL;DR

This paper investigates whether large, pre-trained multilingual language models can effectively handle orthographic and lexical variations in low-resource Occitan dialects without explicit normalization. By fine-tuning mBERT on multi-dialect Occitan data, they found that while explicit normalization isn't always necessary for downstream tasks like POS tagging and dependency parsing, the model still struggles with subtle lexical variations.

What Problem Does This Paper Address?

Natural Language Processing (NLP) pipelines often rely on pre-processing steps like normalization to clean noisy text data, especially for languages with non-standardized writing systems. This is particularly challenging for low-resource languages like Occitan, which lack extensive labeled datasets for normalization. Traditional normalization methods can be computationally expensive and impractical for these languages. This paper explores an alternative: leveraging the power of large, pre-trained multilingual models (like mBERT) and assessing their ability to represent and process dialectal variations without explicit normalization, thereby reducing the pre-processing burden.

Methods

The core of this research involves fine-tuning a pre-trained multilingual BERT (mBERT) model on a dataset of Occitan dialects. The authors then evaluate the fine-tuned model's performance on several tasks:

  • Intrinsic Evaluation:
    • Analogy Computation: Assessing the model's ability to understand semantic and syntactic relationships between words by solving analogy tasks (e.g., "king is to queen as man is to ?"). They used two common metrics: 3CosADD and 3CosMUL.
  • Lexicon Induction: Evaluating how well the model can map words from one dialect to their equivalents in another (specifically, inducing the Lengadocian lexicon from other dialects). This tests the model's understanding of cross-dialectal lexical similarity.
  • Extrinsic Evaluation:
    • Part-of-Speech (POS) Tagging and Universal Dependency (UD) Parsing: Training task-specific layers on top of the fine-tuned mBERT model using the Tolosa Treebank, a multi-dialect Occitan dataset. This assesses the practical utility of the model's representations for downstream NLP tasks.

The experiments were conducted in two main setups: one using data from all four studied dialects for training, and another simulating a more realistic low-resource scenario by training on data from only one dialect (Lengadocian).

Datasets

  • Fine-tuning Data: A combined corpus of 386,552 lines (10,941,124 tokens) from two sources:
  • OcWikiDisc corpus: Compiled from Wikipedia discussion forums in Occitan, known to contain several dialects.
  • WikiMatrix corpus: A parallel sentence corpus mined from Wikipedia pages in 96 languages, from which Occitan data was extracted.
  • Evaluation Datasets:
    • Parallel Lexicon: A newly compiled dataset of over 2,200 entries encompassing four Occitan dialects (Lengadocian, Lemosin, Provençau, and Gascon) for intrinsic evaluations. This dataset was created to explicitly compare model performance across dialects on the same content.
  • Tolosa Treebank: A multi-dialect dataset for Occitan used for extrinsic evaluations (POS tagging and UD parsing).

Key Insights and Ideas

  • Normalization Might Not Be Essential: The study suggests that for low-resource languages with significant dialectal variation, large multilingual models might implicitly handle orthographic and lexical differences without requiring explicit normalization during pre-processing. This is a significant finding for reducing computational costs and complexity in NLP pipelines.
  • Surface Similarity Matters for Transfer Learning: The research hints that while the model can learn from multi-dialect data, it struggles with lexical items that have low surface similarity (i.e., are spelled very differently) even if they are semantically related. This aligns with prior work suggesting that surface similarity plays a crucial role in effective cross-lingual transfer.
  • Dialectal Representation is Uneven: Even after fine-tuning, the model's performance varies across dialects. For instance, it performs better on Lengadocian lexicon induction, likely due to it being better represented in the training data.
  • Intrinsic vs. Extrinsic Evaluation Discrepancies: The model shows poor performance on semantic analogies but better results on syntactic analogies. This suggests that current intrinsic evaluation methods might not fully capture the model's understanding of nuanced semantic relationships, especially across dialects.

Key Results

  • Analogy Computation: The base mBERT model and the fine-tuned model showed poor accuracy on analogy tasks overall. The fine-tuned model saw a slight improvement, particularly for Gascon using the 3CosMUL metric. Semantic analogies yielded 0.0 accuracy for all dialects.
  • Lexicon Induction: Fine-tuning mBERT with multi-dialect Occitan data led to improvements in inducing the Lengadocian lexicon from other dialects. Accuracy ranged from 29.1% (Lemosin) to 40.9% (Provençau). The model performed better when the target and source dialect words had higher surface similarity (lower Levenshtein distance).
  • POS Tagging: Accuracy was relatively high across both training setups (all dialects vs. only Lengadocian). The fine-tuned model showed small improvements over the baseline for Gascon, Lengadocian, and Provençau. Training on all dialects yielded a higher overall accuracy (94.1%) compared to training on only Lengadocian.
  • UD Parsing: Performance was generally lower than POS tagging. Gascon had the highest labeled attachment scores. Similar to POS tagging, performance was best when trained on all four dialects.

Limitations and Future Work

  • Model Specificity: The study only used mBERT. Future work could explore other language models, especially those pre-trained on languages more closely related to Occitan, to see if that improves performance.
  • Data Characterization: The exact dialectal composition and amount of data from each dialect in the fine-tuning corpora (OcWikiDisc and WikiMatrix) are not fully known, making it difficult to definitively attribute performance variations to data distribution.
  • UD Parser Performance: The UD parsers used in this study performed worse than state-of-the-art parsers, potentially limiting the utility of the dialect-specific Occitan UD data.
  • Tokenizer Impact: The authors suggest that the model's tokenizer might contribute to performance detriments when fine-tuned on noisy data. Future work could investigate the overlap between Occitan dialect subtokens.
  • Surface Similarity Enhancement: Future research could explore ways to increase surface similarity between pre-training and fine-tuning data, perhaps by using models pre-trained on related languages or by injecting noise into the pre-training data.

Summary updated

BibTeX
@inproceedings{hopton2024modeling,
  author = {Zachary William Hopton and Noëmi Aepli},
  title = {{Modeling Orthographic Variation in Occitan’s Dialects}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure