TL;DR
Large Language Models (LLMs) excel at high-resource languages, but their performance on endangered languages like Formosan languages remains largely unknown. This paper introduces FORMOSANBENCH, the first benchmark for evaluating LLMs on three Formosan languages across machine translation, ASR, and summarization, revealing a significant performance gap and the urgent need for more inclusive NLP.
What Problem Does This Paper Address?
The incredible success of Large Language Models (LLMs) in Natural Language Processing (NLP) has primarily been demonstrated on languages with abundant digital resources, like English and Mandarin. However, the vast majority of the world's languages, many of which are endangered and spoken by minority communities, are severely underrepresented in NLP research. This creates a critical gap, as LLMs may not be suitable for these communities, potentially exacerbating digital divides. Formosan languages, spoken in Taiwan, are a prime example of such underrepresented languages. They are linguistically rich but endangered, facing challenges like limited high-quality data and unique grammatical structures (e.g., verb-initial syntax, complex voice systems) that make them difficult for current NLP models. This paper aims to bridge this gap by providing a comprehensive benchmark to evaluate LLM capabilities on these languages.
Methods
The authors introduce FORMOSANBENCH, a novel multi-task benchmark designed to evaluate LLMs on three endangered Formosan languages: Atayal, Amis, and Paiwan. The benchmark covers three core NLP tasks:
- Machine Translation (MT): Evaluating translation between Formosan languages and Mandarin.
- Automatic Speech Recognition (ASR): Transcribing spoken Formosan languages.
- Text Summarization: Generating summaries of Formosan language texts.
They systematically evaluate state-of-the-art LLMs using three settings:
- Zero-shot: Models perform tasks without any task-specific training data.
- 10-shot: Models are provided with a few examples (10) of the task in the prompt.
- Fine-tuning: Models are further trained on a small amount of task-specific data.
They also employ an LLM-as-a-judge evaluation for MT and summarization, alongside human evaluation, to assess translation adequacy and summary quality.
Datasets
The paper introduces FORMOSANBENCH, a new benchmark comprising datasets for three Formosan languages (Atayal, Amis, and Paiwan).
- Machine Translation: Approximately 5,000 sentence pairs per language, extracted from the Taiwan Indigenous Languages E-Dictionary and preprocessed for quality.
- Automatic Speech Recognition (ASR): Paired sentence-level audio recordings with transcriptions, sourced from the Klokā Digital Platform for Indigenous Languages.
- Text Summarization: Wikipedia articles in Formosan languages, where introductory sections were paired with the main body content.
These datasets are made publicly available to facilitate future research.
Key Insights and Ideas
- Significant Performance Gap: Even the most advanced LLMs perform poorly on Formosan languages across all evaluated NLP tasks, highlighting their limitations in low-resource, typologically diverse settings.
- Adaptation Offers Limited Gains: While 10-shot learning and fine-tuning provide some improvements, they do not bridge the substantial performance gap, underscoring the need for more targeted adaptation strategies.
- Formosan Languages' Complexity: The unique linguistic features of Formosan languages, such as complex verb morphology and voice systems, pose significant challenges for current LLM architectures.
- ASR Shows Relative Promise: ASR performance, particularly with the MMS model, is relatively better than MT and summarization, though still not at a deployable level.
- Mandarin-to-Formosan Easier: Translating from Mandarin to Formosan languages yields slightly better results than the reverse direction, likely due to Mandarin's prevalence in training data and potential token overlap.
Key Results
- Machine Translation: Zero-shot BLEU scores were consistently near zero for all models, indicating a near-total failure to understand and translate from Formosan languages. Even with fine-tuning, NLLB achieved a maximum BLEU score of 0.1506 (Mandarin to Paiwan), which is still very low.
- Automatic Speech Recognition: The MMS model achieved the lowest Word Error Rates (WER), with scores around 0.522 for Amis and 0.893 for Paiwan in zero-shot. Fine-tuned Whisper significantly reduced WER to around 0.3 for all languages.
- Text Summarization: ROUGE-2 and ROUGE-L scores were generally below 20 for most models in zero-shot, indicating poor summary quality. GPT-40 showed notable improvements with 10-shot learning for Atayal (ROUGE-2: 38.36, ROUGE-L: 50.41), but these gains did not generalize broadly.
Limitations and Future Work
The authors acknowledge that their study is limited to only three Formosan languages. Expanding to more languages and NLP tasks would provide a more comprehensive understanding of low-resource language performance. However, this expansion is constrained by the significant human effort required for data annotation and validation. Future work should focus on developing more targeted strategies and novel architectures to improve LLM performance for typologically diverse, underrepresented languages.
Summary updated