TL;DR
This paper introduces the first benchmark dataset for Kalaallisut (Greenlandic) to Danish Neural Machine Translation (NMT), addressing a significant gap in NLP resources for this low-resource indigenous language. Their findings suggest that even simpler NMT models can outperform sophisticated LLMs in a zero-shot setting, highlighting the need for careful evaluation and specialized approaches for morphologically complex languages.
What Problem Does This Paper Address?
Natural Language Processing (NLP) has seen remarkable advancements, but these often leave behind low-resource languages, particularly those with complex linguistic structures. Kalaallisut, spoken in Greenland, is a prime example. It's a polysynthetic language, meaning words are formed by stringing together many morphemes, leading to extremely long and complex words. This morphological complexity, coupled with a historical lack of standardized benchmarking datasets, has made it challenging to apply modern NLP techniques like Neural Machine Translation (NMT). While some efforts exist for other indigenous languages, a dedicated benchmark for Kalaallisut-Danish NMT was missing, hindering progress and robust evaluation.
Methods
The authors tackled this by first creating a new benchmark dataset. They scraped articles from the public website of Kalaallit Nunaata Radioa (KNR), Greenland's national broadcaster, which publishes content in both Greenlandic and Danish. To create parallel sentence pairs, they employed a straightforward sentence-splitting and alignment algorithm. Documents were split into sentences based on punctuation, and then pairs of articles with an identical number of sentences were assumed to have a one-to-one sentence mapping.
For NMT, they trained custom Recurrent Neural Network (RNN) models using the OpenNMT framework with a PyTorch backend. They experimented with different Byte Pair Encoding (BPE) tokenization strategies, varying the number of merge operations. Additionally, they tested the performance of state-of-the-art Large Language Models (LLMs) like GPT-3.5-turbo and GPT-4 using a zero-shot prompting approach. Evaluation was performed using standard metrics: BLEU and chrF for surface similarity, and BERTScore for semantic similarity.
Datasets
The core contribution is the creation of a new benchmark dataset for Kalaallisut-Danish NMT. This dataset comprises approximately 73,000 parallel sentence pairs, totaling around 1.2 million words in Kalaallisut and 2.1 million words in Danish. A randomly drawn sample of 1,000 sentences was set aside as a test set for model evaluation. This dataset is novel in its focus on the Kalaallisut-Danish language pair and aims to serve as a reliable benchmark for future NMT research in this domain.
Key Insights and Ideas
- Greenlandic is Amenable to NMT: Despite its perceived linguistic uniqueness and morphological complexity, the results suggest that Greenlandic is indeed amenable to NMT, with models achieving reasonable performance, especially when evaluated semantically.
- Simpler Models Outperform LLMs (in Zero-Shot): Surprisingly, a custom Bi-LSTM model significantly outperformed sophisticated LLMs like GPT-3.5-turbo and GPT-4 in a zero-shot setting. This suggests that current LLMs may not be optimally suited for this specific task without further fine-tuning or specialized prompting.
- BPE Merges Impact Performance: Increasing the number of BPE merges generally led to a decrease in performance, indicating that overly aggressive sub-word tokenization might be detrimental for morphologically rich languages like Greenlandic.
- Semantic vs. Lexical Evaluation: Models performed better on semantic similarity metrics (BERTScore) than on surface similarity metrics (BLEU, chrF). This implies that the translations capture the meaning but might differ lexically or stylistically from human references.
Key Results
The paper presents quantitative results in Table 1. The best-performing model was a Bi-LSTM with 5k BPE merges, achieving a BERTScore F1 of 0.74, BLEU of 16, and chrF2 of 32.3. As the number of BPE merges increased, performance generally decreased. The LLMs showed lower scores: GPT-3.5-turbo achieved a BERTScore F1 of 0.64, BLEU of 3, and chrF2 of 25.4, while GPT-4 scored slightly higher with 0.68, 4, and 28.3 respectively.
Limitations and Future Work
The authors acknowledge several limitations. The dataset creation process was "crude" and relied on pragmatic decisions by non-native speakers, suggesting that greater quality control and human intervention are needed for future iterations. Additionally, the KNR texts themselves have limitations, with some being literal translations from Danish and potentially containing grammatical errors or Danish influence, which might affect translation quality.
Future work includes experimenting with more sophisticated neural architectures like transformers and exploring pre-trained multilingual embeddings (e.g., mT5, mBART). They also envision this work as a stepping stone towards practical applications like speech-to-text models and improved linguistic analysis, and importantly, fostering collaboration with researchers working on other Inuit-Yupik-Unangan languages to pool resources and advance language technology in the region.
Summary updated