TL;DR
This paper investigates how to build effective Part-of-Speech (POS) tagging training sets for under-resourced languages, comparing in-context learning with large language models (LLMs) against traditional active learning (AL) and random sampling. It finds that for languages where data can be shared via API, a small prompt with GPT-4.1-mini can achieve high performance, while for data-sensitive communities, AL offers a more efficient path to good results than random sampling.
What Problem Does This Paper Address?
Building Natural Language Processing (NLP) tools for languages with limited resources, especially Indigenous and endangered languages, is a significant challenge. This is due to a lack of financial support, fewer data sources, and difficulties in manual annotation. The authors focus on Part-of-Speech (POS) tagging as a case study to explore how to construct training sets that yield generalizable model performance, particularly when resources are scarce. They question how much data is needed and, more importantly, what kind of data is most effective for training these models.
Methods
The paper compares three main approaches for training set construction:
- In-Context Learning (ICL) with LLMs: This method uses large language models like GPT-4.1-mini. The authors provide a small number of examples (around 1,000 tokens) directly within the prompt to guide the LLM's output for POS tagging. This approach is explored for its potential to achieve good results with minimal data, assuming data can be shared via an API.
- Active Learning (AL): This is an iterative process where a model is trained on an initial small set of data. It then identifies the most "uncertain" data points from a larger pool, which are then manually annotated and added to the training set. This continues until a desired performance level is reached. The authors simulate this process using existing annotations.
- Random Sampling: This is a baseline method where data points are randomly selected from the available pool to build the training set, without any intelligent selection strategy.
The authors also employ growth curve modeling to statistically analyze the learning curves generated by AL and random sampling, providing a more rigorous comparison than visual inspection alone.
Datasets
The study utilizes the Universal Dependencies (UD) project, a collection of treebanks (annotated text corpora) for many languages. They used data from 60 languages across 12 language families. For in-context learning, they used approximately 1,000 randomly sampled tokens as prompt examples for each treebank. For active learning and random sampling, they expanded their training data pool to include up to 112 treebanks.
Key Insights and Ideas
- LLMs as a "Quick Win" (with caveats): For languages where data sharing is not an issue, using a powerful LLM like GPT-4.1-mini with just 1,000 prompt examples can deliver surprisingly good POS tagging performance, often on par with models trained on much larger datasets. This suggests LLMs can be a valuable tool for initial data annotation.
- Data Sensitivity Matters: The paper highlights that for many Indigenous communities, data sovereignty is paramount. In such cases, using external APIs like GPT-4.1-mini might be unethical. This underscores the importance of methods like AL that can be run locally.
- AL's Efficiency Edge: While both AL and random sampling can eventually reach similar performance levels, AL consistently demonstrates a faster learning rate. This means it can achieve a comparable performance with fewer training data points compared to random sampling, making it more efficient for resource-constrained scenarios.
- Syntactic Diversity vs. Word Distribution: The study found that for POS tagging, a more variable distribution of words within a specific tag (e.g., many different words tagged as NOUN) generally leads to better model performance. Conversely, less diverse syntactic environments for a POS tag also tend to yield better results.
Key Results
- GPT-4.1-mini Performance: For most of the 60 languages studied, GPT-4.1-mini achieved weighted F1 scores above 0.83, with many reaching 0.90 or higher, using only 1,000 prompt tokens. This performance was often comparable to or better than AL iterations requiring thousands of tokens.
- Active Learning Efficiency: For languages where data cannot be shared via API, AL using a statistical model (CRF) was found to be more efficient than random sampling. For example, in the UD_Irish-IDT treebank, AL reached an F1 score of 0.85 with 4,500 tokens, while random sampling required more. AL models also approached their maximum performance faster.
- Cost-Effectiveness: The authors note that API calls for GPT-4.1-mini cost less than $4 USD per treebank, making it a more economical option than manually annotating thousands of tokens for AL, assuming data sharing is permissible.
Limitations and Future Work
- LLM Training Data Bias: The authors acknowledge that the UD treebanks used might have been part of GPT-4.1-mini's training data, potentially inflating its performance. They also note that the languages studied were not truly "low-resource" by some definitions, and performance on genuinely endangered languages might be worse.
- Fixed Prompt Size: The study used a fixed 1,000-token prompt for LLMs. Future work could explore if smaller or larger prompts yield different results.
- Simulated AL: The active learning process was simulated using existing annotations. Future research should investigate the real-world impact of AL on annotator effort and efficiency.
- Downstream Task Impact: The paper focuses on POS tagging performance itself. Future work could explore how improvements in POS tagging translate to better performance on downstream NLP tasks.
- Broader NLP Tasks: The authors suggest extending their methodology to other NLP tasks relevant to documenting endangered languages, such as automatic speech recognition and morphological segmentation.
- Diversity Sampling: Exploring diversity sampling techniques within active learning is also a promising future direction.
Summary updated