TL;DR
This paper tackles the challenge of normalizing Estonian dialectal text to standard Estonian, a task hindered by a lack of readily available parallel data. The authors ingeniously leverage a Finnish dialect generator to create synthetic Estonian dialect data, significantly improving the performance of their BERT-based normalization model.
What Problem Does This Paper Address?
Natural Language Processing (NLP) tools often struggle with non-standard language variations, and Estonian is no exception. While standard Estonian has good NLP support, its numerous dialects are severely under-resourced. This means that informal communication, like messages or online discussions, often falls outside the capabilities of current NLP models. Dialect normalization aims to bridge this gap by converting dialectal text into a standard form, allowing existing NLP tools to be applied. However, a significant hurdle for Estonian dialect normalization is the scarcity of parallel corpora – datasets containing dialectal sentences aligned with their standard Estonian counterparts.
Methods
The core of this paper's contribution lies in its innovative approach to generating synthetic training data. The authors first crawl a small corpus of naturally occurring parallel dialectal and standard Estonian sentences from online dialect dictionaries. Crucially, they then employ a Finnish dialect generator model, previously developed for Finnish dialects, to "dialectalize" standard Estonian sentences. This involves a clever adaptation process: Estonian 'ü' is temporarily converted to Finnish 'y', and 'õ' is mapped to 'ö' before generation, with post-processing to restore the correct Estonian characters. They also implement a simple post-processing step to match the voicing of plosives between the generated dialect and the standard Estonian input.
For the normalization task itself, the authors train BERT-based sequence-to-sequence models. They experiment with three configurations: one trained solely on the crawled Estonian dialect data, another on the synthetically generated Finnish-dialect-like Estonian data, and a third trained on both.
Datasets
- Crawled Estonian Dialect Data: A small corpus of parallel dialectal and standard Estonian sentences extracted from online dialect dictionaries (Mulgi, Kihnu, and Hiiu dialects). This dataset resulted in 14,510 aligned sentences after cleaning and filtering.
- Synthetic Estonian Dialect Data: Generated by applying a Finnish dialect generator to standard Estonian sentences from the Universal Dependencies (UD) treebanks. This process yielded over 336,000 synthetic sentence pairs.
- Estonian Universal Dependencies (UD) Treebanks: Used as the source for generating synthetic dialectal data.
Key Insights and Ideas
- Leveraging "Neighboring" Languages: The success of using a Finnish dialect generator for Estonian highlights the potential of cross-lingual transfer learning, even between closely related languages, when direct resources are scarce.
- Synthetic Data is Key: The paper strongly suggests that synthetic data, even if not perfectly representative of true dialects, can significantly boost performance in low-resource scenarios.
- Combined Data is Superior: Training a model on both real (though limited) dialectal data and synthetic data yields the best results, indicating a synergistic effect.
- Normalization Strategy Matters: The authors note that the strict normalization strategy used in the original dialect dictionaries (sometimes replacing words entirely) makes the task more challenging than in cases where only orthography and morphology are adjusted.
Key Results
The BERT-based normalization model trained on both the crawled Estonian data and the synthetically generated data achieved a Word Error Rate (WER) of 55.25% with an accuracy of 0.471. This represents a significant improvement compared to models trained on only Estonian data (WER 77.74%) or only synthetic data (WER 73.84%). Even the baseline (no normalization) had a very high WER of 74.09%. While the results are not perfect, the data augmentation drastically improved performance.
Limitations and Future Work
The authors acknowledge that even their best model makes mistakes about half the time, and the WER is still relatively high. This is partly attributed to the original normalization strategy in the source data, which sometimes involved more than just orthographic and morphological adjustments. They also note that the synthetic data, while helpful, doesn't perfectly capture the nuances of real Estonian dialects. Future work could involve exploring more sophisticated methods for generating synthetic data, potentially by incorporating more linguistic knowledge or by experimenting with different Finnish dialects to find a closer match. Further research into the specific challenges posed by the original normalization strategy is also suggested.
Summary updated