Abstract
While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.
The "Hard Compound" Problem
In many languages, new words are frequently formed by joining existing ones (e.g., "bridesmaid"). Decompounding—identifying these constituents—is vital for search, translation, and lexicography. However, modern Large Language Models (LLMs) struggle with this task due to their reliance on subword tokenization (like BPE or SentencePiece).
These tokenizers are linguistically uninformed, splitting words based on statistical frequency. When a subword boundary misaligns with a linguistic constituent boundary, it creates a "hard compound." For example, splitting "windsurfing" into winds, urf, and ing mangles the internal representation, making it nearly impossible for even powerful models like FLAN-UL2 to correctly identify the base words "wind" and "surfing."
Methodology: A Two-Stage Approach
To address these failures, we propose a dedicated byte-level decompounding model using ByT5, which avoids the pitfalls of subword tokenization by operating on raw bytes.
- Stage 1: Self-Supervised Hyphen Prediction. We leverage hyphens in web text (from mC4) as high-precision indicators of compound boundaries. Training a model to restore these hyphens allows it to learn the "rhythm" of compound formation without human labels.
- Stage 2: Supervised Fine-Tuning. We fine-tune on a newly curated Wiktionary dataset to handle compound normalization (e.g., mapping "bridesmaid" to "bride" + "maid").
CompoundPiece Tokenization
We further introduce CompoundPiece, a tokenizer that uses our decompounding model to pre-segment compounds during its own creation. This ensures that subword boundaries align with linguistic boundaries, effectively converting "hard" compounds into "easy" ones.
Data: A Massive 56-Language Benchmark
A primary contribution of this work is a new dataset of 255,000 words across 56 languages, curated from Wiktionary. This includes both positive (compound) and negative (non-compound) examples, providing the first large-scale, multi-language benchmark for the task.
Results
Our proposed ByT5 model (Stage 1 + 2) achieved a 94.8% macro accuracy across all languages, significantly outperforming prior language-specific tools. Even in the unsupervised setting (Stage 1 only), our model outperformed the previous state-of-the-art (SECOS) by 13.9%.
Crucially, our models show a massive performance gap over standard LLMs on "hard compounds." While models like FLAN T5 XXL improve with more shots on "easy" compounds, they remain stagnant and perform poorly on compounds where tokenization is unfavorable.
Conclusion
CompoundPiece demonstrates that linguistically informed tokenization is essential for handling the morphological complexity of global languages. By aligning subword boundaries with actual word constituents, we can significantly improve the performance of language models on fundamental linguistic tasks.
Summary updated