TL;DR
This paper investigates how the amount of preceding text (context) influences text tokenization, a crucial preprocessing step for many NLP tasks. They found that the optimal tokenization strategy is language-dependent, and while context generally improves performance, its impact is more subtle for word segmentation than for evaluating different tokenization methods.
What Problem Does This Paper Address?
Tokenization, the process of breaking down text into meaningful units (tokens), is a fundamental step in Natural Language Processing (NLP). While modern language models can handle very long contexts, the specific role of this context in tokenization itself remains underexplored. Many tokenization methods are driven by language models, creating a potential chicken-and-egg problem. This paper aims to clarify how the amount of preceding text impacts the effectiveness of different tokenization strategies across various writing systems and how it affects the evaluation of these strategies.
Methods
The authors designed two experiments to tackle this question.
Experiment 1: Comparing Tokenization Strategies with Varying Context They compared three segmentation strategies:
- Word segmentation: Using pre-tokenized text as is.
- Morfessor: An unsupervised morphological segmentation algorithm.
- BERT tokenizer: Subword tokenization from pre-trained Huggingface models (BPE-based).
For each strategy and language (English, Turkish, Chinese), they trained n-gram language models with increasing context sizes (from unigram to 5-gram) and evaluated them using bits-per-sentence (BPS), a measure of language model surprisal. This allowed them to see how the "optimality" of a segmentation strategy changes as more context is considered.
Experiment 2: Contextualized Word Boundary Detection This experiment focused specifically on word segmentation. They simulated word segmentation by removing all punctuation and whitespace from sentences. Then, they used language models of increasing order (uni-, bi-, and trigram) to reconstruct word boundaries. This was framed as a shortest path problem on a graph, where the path's score reflects the language model's probability. They evaluated the performance using accuracy, F1-score, recall, and precision.
Datasets
The experiments were conducted on the OpenSubtitles2016 corpus, a large collection of movie subtitles. The corpus was adapted to include an overlapping set of movies across English, Turkish, and simplified Chinese. The data was split into 90% for training and 10% for testing.
Key Insights and Ideas
- Language-Specific Tokenization: The optimal way to segment text for language modeling is not universal; it depends heavily on the specific characteristics of the language and its writing system.
- Context Matters, But Differently: While increasing context generally improves language modeling performance (reduces surprisal), its impact on evaluating different tokenization strategies can be significant, even reversing rankings. However, for the specific task of detecting word boundaries, the role of context was more subtle.
- Subword vs. Word Segmentation: For agglutinative (Turkish) and logographic (Chinese) languages, subword segmentations (Morfessor, BERT) benefit more from increased context than traditional word segmentation. English showed a more mixed result, with word segmentation performing comparably.
- Word Boundary Detection is Robust: Even with limited context (unigram models), word boundary recognition can achieve high accuracy, especially when the language model is trained on a sufficiently large and representative corpus.
Key Results
Experiment 1: * For English, word segmentation consistently scored slightly better or on par with subword methods, even with increased context. * For Turkish and Chinese, the ranking reversed: subword segmentations (Morfessor and BERT) outperformed word segmentation as context size increased, particularly around 20% of sentence length. BERT was the most optimal for Chinese. * Subword vocabularies were significantly smaller (an order of magnitude) than word vocabularies.
Experiment 2: * The impact of increasing language model order (from unigram to bigram/trigram) on word boundary recognition accuracy was generally small, especially compared to the effect of increasing the corpus size used for training the language model. * Accuracy approached 100% for all languages when trained on large corpora. * Bigram models showed a slight improvement over unigram models, particularly in recall, suggesting that ignoring context can lead to undersegmentation. However, the benefit of trigram models was minimal.
Limitations and Future Work
- Language Model Type: The study primarily used count-based n-gram language models. The authors acknowledge that the extent to which these results generalize to other types of language models, such as neural network-based models, is unclear.
- Future Work: The authors suggest exploring higher-order segmentation in future work, particularly for tasks like morphological segmentation or syllabification. They also note the potential for further investigation into how context influences the evaluation of tokenizers themselves.
Summary updated