Abstract
The role of subword segmentation in relation to capturing morphological patterns in LLMs is currently not well explored. Ideally, one would train models like GPT using various segmentations and evaluate how well word meanings are captured. Since this is not computationally feasible, we group words according to their segmentation properties and compare how well a model can solve a linguistic task for these groups. We study two criteria: (i) adherence to morpheme boundaries and (ii) the segmentation consistency of the different inflected forms of a lemma. We select word forms with high and low values for these criteria and carry out experiments on GPT-4o’s ability to capture verbal inflection for 10 languages. Our results indicate that in particular the criterion of segmentation consistency can help to predict the model’s ability to recognize and generate the lemma from an inflected form, providing evidence that subword segmentation is relevant.
Introduction: The Morphological Gap
Large Language Models (LLMs) do not process text as whole words. Instead, they utilize tokenizers—most commonly Byte Pair Encoding (BPE)—to decompose text into subword units. These tokenizers are typically optimized for frequency statistics rather than linguistic structure. This often results in a "morphological gap" where the internal representations of related words (e.g., a root and its inflections) do not align with their linguistic components.
As shown in the table above, GPT's segmentation of German verbs often ignores the boundaries between stems and suffixes. This raises a critical question: if a model sees different internal representations for the same root word across its various forms, can it still recognize their shared semantic meaning? This is particularly challenging for morphologically rich languages like Finnish or Hungarian.
Methodology
To investigate this, we evaluated GPT-4o across 10 languages (English, French, German, Spanish, Italian, Portuguese, Finnish, Swedish, Czech, and Hungarian). We focused on two primary criteria for segmentation quality:
- Adherence to Morpheme Boundaries: Does the model split the word exactly where the "stem" ends and the "suffix" begins?
- Segmentation Consistency: Does the model split the stem of a word the same way across different inflections? This was measured using the Overlap Coefficient, which quantifies the subword overlap between an inflected form and its base lemma.
We categorized segmentations into several types to analyze their impact on performance.
The model was tested on two downstream tasks: Lemma Prediction (identifying the base form from an inflection) and Inflection Generation (producing the correct inflected form from a lemma and morphological tags). Data was sourced from MorphyNet and word frequencies were calculated using Wikipedia dumps.
Key Insights: Consistency Over Purity
Our analysis revealed several surprising trends regarding how LLMs handle morphology:
- Consistency Trumps Linguistic Purity: Whether a subword split aligns perfectly with a linguistic boundary (e.g., stem/suffix) has negligible impact on performance. However, internal consistency is vital. If the model segments the stem inconsistently across a word's paradigm, its ability to perform morphological reasoning drops significantly.
- The "First Token" Anchor: Performance is notably higher when the first subword token of an inflected form is identical to that of its lemma. This "anchor" helps the model maintain semantic connectivity.
- Frequency as a Buffer: For high-frequency words, the model likely relies on memorization, making segmentation quality less critical. For rare or unseen words, however, consistent segmentation becomes the primary driver for successful reasoning.
- Language Sensitivity: The negative impact of inconsistent segmentation is most pronounced in languages with complex morphology, such as Hungarian and Finnish.
Key Results
The results underscore the importance of consistency. In Hungarian, for low-frequency words, GPT-4o correctly identified the lemma 79% of the time when segmentation was consistent, but this dropped to 36% when it was inconsistent. Conversely, in 9 out of 10 languages, there was no statistically significant difference in performance based on whether words were split at exact morpheme boundaries.
While one-shot prompting (providing a single example) helped improve performance for inconsistently segmented words, it failed to fully bridge the gap compared to those with consistent splits.
Limitations and Future Work
This study primarily focuses on "surface" or concatenative splits. It does not fully address non-concatenative morphology, such as German umlaut changes (Apfel to Äpfel) or English irregulars (go to went). Additionally, the use of English-centric prompts for all languages may introduce some bias.
Future research could explore training models with "consistency-optimized" tokenizers to determine if improving morphological alignment from the ground up enhances the reasoning capabilities of LLMs across diverse languages.
Summary updated