CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models

Benjamin Minixhofer, Jonas Pfeiffer, Ivan Vulic

Abstract

Compounding—the process of joining two or more words to create new lexical units—is a fundamental linguistic process in many languages, such as German, Dutch, and Finnish. Despite its prevalence, Large Language Models (LLMs) often struggle with decompounding, the task of splitting these words into their constituents. This difficulty is largely driven by subword tokenization strategies that split compounds into linguistically nonsensical units. In this work, we introduce a massive dataset of 255k words across 56 languages and a novel two-stage training framework for decompounding. Our approach, which leverages byte-level models and self-supervised hyphen prediction, significantly outperforms existing tools. Furthermore, we introduce CompoundPiece, a tokenization method that aligns subword boundaries with linguistic constituents, leading to improved performance on downstream tasks.

Introduction: The "Hard Compound" Problem

Modern Large Language Models typically rely on subword tokenization algorithms like Byte Pair Encoding (BPE) or SentencePiece. While efficient for compressing vocabulary, these methods are purely statistical and often ignore linguistic structure. For example, a tokenizer might split the compound "windsurfing" into fragments like winds, ur, f, and ing. This obscures the semantic relationship between the root words "wind" and "surfing."

We define these cases as "hard compounds": words where the subword boundaries do not align with the linguistic boundaries of the constituents. When a model encounters a hard compound, its ability to generalize and understand the word's meaning is severely compromised. This paper addresses this bottleneck by providing the data and methodology needed to improve decompounding at scale.

A New Multilingual Benchmark

A major hurdle in decompounding research has been the lack of diverse, large-scale data. Previous studies focused primarily on a few productive languages like German. To bridge this gap, we extracted a dataset of 255,000 words across 56 languages from Wiktionary.

Unlike previous benchmarks, our dataset includes both compound words and "negatives" (non-compound words). This allows for a more robust evaluation of a model's ability to identify when not to split a word, a critical requirement for real-world applications. Additionally, we utilize the mC4 corpus for self-supervised training, leveraging approximately 25 million hyphenated words as natural indicators of compound boundaries.

Methodology

We propose a two-stage training procedure using ByT5, a byte-level model. By processing raw characters rather than subwords, ByT5 avoids the very tokenization pitfalls we aim to solve.

Stage 1: Self-Supervised Hyphen Prediction

We observed that hyphens in web text often act as high-precision proxies for compound boundaries. We trained a model to "restore" hyphens in words where they had been intentionally removed. This self-supervised objective allows the model to learn the structural logic of compounds across dozens of languages without requiring human-labeled data.

Stage 2: Supervised Normalization

In the second stage, we fine-tune the model on our Wiktionary dataset. The goal here is "compound normalization": the model must not only identify the split points but also return the constituents to their base forms (e.g., converting the German Häusermarkt to Haus and Markt).

CompoundPiece Tokenization

Finally, we integrate our best decompounding model into the tokenization pipeline itself. CompoundPiece acts as a pre-tokenizer; before standard SentencePiece processing occurs, our model splits compounds into their constituents. This ensures that the resulting subwords are linguistically meaningful.

Key Insights and Results

Our evaluation revealed several critical findings regarding the current state of LLMs and the efficacy of our proposed methods:

  1. The Subword Trap: We found that state-of-the-art LLMs like FLAN-T5 and UL2 perform reasonably well on "easy" compounds but their accuracy drops to near zero on "hard" compounds. This confirms that subword tokenization is a primary bottleneck for linguistic understanding in compounds.
  2. Superior Decompounding: Our fine-tuned ByT5 model (Stage 1+2) outperformed all prior language-specific decompounding tools across all tested datasets, including GermaNet and AuCoPro. Our self-supervised models alone outperformed previous unsupervised state-of-the-art models by an average of 13.9%.
  3. Improved Tokenization: CompoundPiece significantly improved tokenization quality, reducing the percentage of "hard compounds" from 27.1% to 9.7% in monolingual settings.
  4. Downstream Performance: Multilingual T5 models trained with CompoundPiece showed a 5.5% improvement in compound normalization tasks compared to standard SentencePiece models, despite maintaining the same parameter count and training steps.

Conclusion

By aligning tokenization with linguistic reality, CompoundPiece offers a path toward more robust and capable multilingual models. While the availability of supervised data in Wiktionary remains a constraint for extremely low-resource languages, our self-supervised hyphen-prediction strategy provides a powerful alternative. Future work will explore the integration of decompounding into broader tasks such as Machine Translation and Information Retrieval, where the semantic clarity of constituents is paramount.

Summary updated

Abstract

While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.
BibTeX
@inproceedings{minixhofer2023compoundpiece,
  author = {Benjamin Minixhofer and Jonas Pfeiffer and Ivan Vulic},
  title = {{CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=xapBkUt0yf},
}
Figure