Abstract
Compounding—the process of joining two or more words to create new lexical units—is a fundamental linguistic process in many languages, such as German, Dutch, and Finnish. Despite its prevalence, Large Language Models (LLMs) often struggle with decompounding, the task of splitting these words into their constituents. This difficulty is largely driven by subword tokenization strategies that split compounds into linguistically nonsensical units. In this work, we introduce a massive dataset of 255k words across 56 languages and a novel two-stage training framework for decompounding. Our approach, which leverages byte-level models and self-supervised hyphen prediction, significantly outperforms existing tools. Furthermore, we introduce CompoundPiece, a tokenization method that aligns subword boundaries with linguistic constituents, leading to improved performance on downstream tasks.
Introduction: The "Hard Compound" Problem
Modern Large Language Models typically rely on subword tokenization algorithms like Byte Pair Encoding (BPE) or SentencePiece. While efficient for compressing vocabulary, these methods are purely statistical and often ignore linguistic structure. For example, a tokenizer might split the compound "windsurfing" into fragments like winds, ur, f, and ing. This obscures the semantic relationship between the root words "wind" and "surfing."
We define these cases as "hard compounds": words where the subword boundaries do not align with the linguistic boundaries of the constituents. When a model encounters a hard compound, its ability to generalize and understand the word's meaning is severely compromised. This paper addresses this bottleneck by providing the data and methodology needed to improve decompounding at scale.
A New Multilingual Benchmark
A major hurdle in decompounding research has been the lack of diverse, large-scale data. Previous studies focused primarily on a few productive languages like German. To bridge this gap, we extracted a dataset of 255,000 words across 56 languages from Wiktionary.
Unlike previous benchmarks, our dataset includes both compound words and "negatives" (non-compound words). This allows for a more robust evaluation of a model's ability to identify when not to split a word, a critical requirement for real-world applications. Additionally, we utilize the mC4 corpus for self-supervised training, leveraging approximately 25 million hyphenated words as natural indicators of compound boundaries.
Methodology
We propose a two-stage training procedure using ByT5, a byte-level model. By processing raw characters rather than subwords, ByT5 avoids the very tokenization pitfalls we aim to solve.
Stage 1: Self-Supervised Hyphen Prediction
We observed that hyphens in web text often act as high-precision proxies for compound boundaries. We trained a model to "restore" hyphens in words where they had been intentionally removed. This self-supervised objective allows the model to learn the structural logic of compounds across dozens of languages without requiring human-labeled data.
Stage 2: Supervised Normalization
In the second stage, we fine-tune the model on our Wiktionary dataset. The goal here is "compound normalization": the model must not only identify the split points but also return the constituents to their base forms (e.g., converting the German Häusermarkt to Haus and Markt).
CompoundPiece Tokenization
Finally, we integrate our best decompounding model into the tokenization pipeline itself. CompoundPiece acts as a pre-tokenizer; before standard SentencePiece processing occurs, our model splits compounds into their constituents. This ensures that the resulting subwords are linguistically meaningful.
Key Insights and Results
Our evaluation revealed several critical findings regarding the current state of LLMs and the efficacy of our proposed methods:
- The Subword Trap: We found that state-of-the-art LLMs like FLAN-T5 and UL2 perform reasonably well on "easy" compounds but their accuracy drops to near zero on "hard" compounds. This confirms that subword tokenization is a primary bottleneck for linguistic understanding in compounds.
- Superior Decompounding: Our fine-tuned ByT5 model (Stage 1+2) outperformed all prior language-specific decompounding tools across all tested datasets, including GermaNet and AuCoPro. Our self-supervised models alone outperformed previous unsupervised state-of-the-art models by an average of 13.9%.
- Improved Tokenization: CompoundPiece significantly improved tokenization quality, reducing the percentage of "hard compounds" from 27.1% to 9.7% in monolingual settings.
- Downstream Performance: Multilingual T5 models trained with CompoundPiece showed a 5.5% improvement in compound normalization tasks compared to standard SentencePiece models, despite maintaining the same parameter count and training steps.
Conclusion
By aligning tokenization with linguistic reality, CompoundPiece offers a path toward more robust and capable multilingual models. While the availability of supervised data in Wiktionary remains a constraint for extremely low-resource languages, our self-supervised hyphen-prediction strategy provides a powerful alternative. Future work will explore the integration of decompounding into broader tasks such as Machine Translation and Information Retrieval, where the semantic clarity of constituents is paramount.
Summary updated