Introduction
In modern natural language processing, the distinction between content units (words with heavy semantic meaning like "moose" or "run") and function units (grammatical markers like "the," "of," or tense suffixes) is often overlooked. However, for low-resource languages where data is scarce, identifying this "grammar glue" is vital for tasks like cross-lingual transfer and syntactic abstraction. This paper investigates whether standard subword tokenizers, specifically Byte-Pair Encoding (BPE), can automatically isolate a language's functional lexicon without annotated data.
Methodology: Productivity and the 50-Merge Hypothesis
The authors propose using BPE to identify function units based on their productivity—the frequency with which a unit co-occurs with a wide variety of other units. Productivity is measured in two ways:
- Word-level: The number of unique words containing a specific subword.
- Trigram-level: The number of unique sequences of three subwords containing the unit, which helps identify whole-word function units like "and."
While typical NLP applications use thousands of BPE merges, the authors hypothesize that a much smaller number of merges (50–350) is optimal for isolating functional units. A standardized productivity threshold ($\zeta$) is then applied to classify subwords as functional or content-based.
Experimental Setup
The study utilizes data from the SIGMORPHON 2023 Shared Task on Interlinear Glossing, covering seven typologically diverse languages: Arapaho, Gitksan, Uspanteko (North America); Lezgi, Tsez (Eurasia); Nyangbo (Africa); and Natügu (Papunesia). These datasets provide "true labels" from linguists, allowing for the verification of the unsupervised BPE method.
Key Results and Discussion
The 50-Merge "Sweet Spot"
The most significant finding is that 50 BPE merges is the optimal setting for identifying function units across all seven languages. As the number of merges increases, creating longer and more specific subwords, the ability to distinguish function from content diminishes.
Consistency Across Languages
Regardless of a language's morphological type—whether isolating or polysynthetic—high productivity remains a reliable signal for functional status. Furthermore, the optimal productivity threshold ($\zeta$) was remarkably consistent across the diverse language set, suggesting the method can be applied to unseen languages with minimal tuning.
Impact on Downstream Tasks
Integrating BPE-50 tokenization into downstream models showed clear benefits for Automatic Interlinear Glossing. The method improved the performance of Statistical Machine Translation (SMT) and transformer-based models (DeepSPIN-3) compared to standard character-level tokenization. While the method primarily identifies the most frequent function units (the "heavy hitters"), these represent the vast majority of actual tokens in a text.
Conclusion and Future Work
This research demonstrates that simple BPE configurations can effectively "smoke out" the functional lexicon of a language. Future work may explore how to provide these identified units with specialized treatment during model training, such as unique embedding initializations, to further enhance low-resource NLP. Additionally, testing these metrics on non-alphabetic scripts remains an open area for investigation.
Summary updated