TL;DR
This paper introduces Morphemo, a novel statistical algorithm for morphological segmentation, and compares its performance to Large Language Models (LLMs) on the under-resourced Bribri language. Morphemo excels at segmenting complex words, outperforming LLMs in this crucial aspect of linguistic documentation.
What Problem Does This Paper Address?
Documenting Indigenous languages often involves a painstaking process of interlinearized annotation, where words are broken down into their constituent morphemes. This task, known as morphological segmentation, is particularly challenging for low-resource languages due to the scarcity of data. Existing methods, including unsupervised and semi-supervised approaches, often struggle with the complexity of these languages, while LLMs, though powerful, can be overly conservative or prone to errors. This paper aims to bridge this gap by developing and evaluating a statistical method that can effectively segment morphologically rich, under-resourced languages, and by comparing its performance against state-of-the-art LLMs.
Methods
The authors introduce Morphemo, a semi-supervised, N-gram-based algorithm designed for morphological segmentation in low-resource settings. Morphemo uses Bayesian inference to estimate the likelihood of a morpheme boundary between characters in a word, considering N-grams before and after the potential boundary, as well as the current number of assigned morpheme boundaries. This approach is dual-forward and backward-facing to capture intuitions about morpheme-boundary relationships.
For comparison, the paper evaluates:
- Byte-Pair Encoding (BPE): A standard unsupervised method.
- Morfessor: A semi-supervised method that uses pre-labeled Bribri words for training.
- Large Language Models (LLMs): Specifically, Claude 3.7 Sonnet, evaluated in three conditions:
- Zero-shot: The LLM is given only the words to segment.
- Few-shot: The LLM is provided with unlabeled training data and labeled examples.
- Few-shot plus unlabeled: The LLM receives labeled training data, unlabeled training data, and a large amount of additional unlabeled monolingual Bribri text.
Datasets
The primary dataset used is a corpus of Bribri language text.
- Labeled Data: A set of 1410 manually segmented Bribri words from the Universal Dependencies TreeBank, derived from oral corpora and textbooks. This data was split into 80% for training and 20% for testing, with the procedure repeated 20 times for robust evaluation.
- Unlabeled Data: A large corpus of unlabeled monolingual Bribri text from the AmericasNLP machine translation corpus (85816 words). This was used to supplement the training of Morphemo and in the "Few shot plus unlabeled" LLM condition.
Key Insights and Ideas
- Morphemo's Strength in Complexity: The most significant finding is Morphemo's superior performance in segmenting multimorphemic words, where LLMs tend to be too conservative.
- LLM Conservatism: LLMs, particularly Claude 3.7, exhibit a conservative segmentation behavior, which benefits them on monomorphemic words but hinders their performance on more complex structures.
- Statistical Methods Remain Relevant: Despite the rise of LLMs, statistical methods like Morphemo continue to be highly effective and even outperform LLMs in specific, crucial tasks like segmenting morphologically rich languages.
- Data Trade-offs: The experiments on varying data sizes reveal that while unlabeled data helps, labeled data is crucial for improving the segmentation of multimorphemic words. There's also a trade-off between aggressiveness and accuracy, where Morphemo becomes more aggressive with more data, improving multimorphemic segmentation at the cost of slightly reduced monomorphemic performance.
Key Results
- Overall Performance: Morphemo achieved a strong average F1 score of 67.7, outperforming BPE (53.6) and Morfessor (58.6).
- Multimorphemic Words: Morphemo significantly outperformed LLMs in segmenting multimorphemic words, achieving an F1 of 59.6, compared to Claude 3.7 Few Shot (57.2) and Claude 3.7 Few Shot + Unlabeled (42.4).
- Monomorphemic Words: LLMs, especially Claude 3.7 Few Shot (85.5) and Few Shot + Unlabeled (87.5), showed higher accuracy on monomorphemic words due to their conservative nature. Morphemo achieved 70.9 on these.
- LLM Performance: Claude 3.7 Few Shot achieved the highest overall F1 score (78.0), largely driven by its strong performance on monomorphemic words.
- Training Efficiency: Morphemo demonstrated a fast training time (1.42 seconds) and lower computational requirements compared to LLMs.
Limitations and Future Work
- Written vs. Oral Language: The current models are trained on written Bribri and cannot process oral narratives, which are more common for many speakers.
- Orthographic Variation: Bribri lacks a single standardized orthography, and the models were trained on a specific set, potentially limiting applicability to other orthographic variations.
- LLM Data Sovereignty: The use of commercial LLMs raises concerns about data sovereignty for communities wishing to process restricted data.
- Future Work: The authors suggest combining statistical and LLM approaches, potentially by informing LLMs with typological properties of languages. They also propose testing Morphemo against other algorithms and LLMs, and exploring rule-based segmentation tools or locally-run software for languages where data cannot be written or processed by commercial entities. Further investigation into how Morphemo's behavior might adapt to languages with higher morpheme-per-word ratios is also planned.
Summary updated