Abstract
Traditional Natural Language Processing (NLP) pipelines often rely on fixed, pre-defined tokenization steps, which can be suboptimal for specific downstream tasks. This paper introduces a method to jointly learn text tokenization and classification by stochastically sampling various segmentations during training. By moving away from a "one-best" deterministic approach, the proposed model gains robustness and achieves superior performance in sentiment analysis and topic detection across Japanese, Chinese, and English.
Introduction
In NLP, defining a "word" is often treated as a solved preprocessing step. While English uses spaces as explicit boundaries, languages like Chinese and Japanese require separate tokenizers to segment text before it reaches a neural model. However, a fixed tokenizer may not be optimal for the task at hand. In "Stochastic Tokenization with a Language Model for Neural Text Classification" (ACL 2019), Hiraoka et al. argue that tokenization should be a learned, task-aware process rather than a static preprocessing step.
The Problem with Fixed Tokenization
Most neural text classifiers depend on a fixed vocabulary. In unsegmented languages, this necessitates tools like morphological analyzers or subword algorithms (e.g., Byte Pair Encoding) to be run beforehand. This pipeline approach suffers from two main flaws:
- Ambiguity: Multiple valid segmentations exist, and a preset tokenizer might choose one that obscures task-relevant information.
- Rigidity: Models trained on a single segmentation become brittle and struggle with out-of-vocabulary words or informal phrasing.
Methodology: Joint Learning and Stochastic Sampling
The authors propose a joint training framework combining a Language Model (LM) for tokenization and a Neural Network for classification.
The core innovation lies in Stochastic Sampling. Instead of the single most likely segmentation, the model uses a nested unigram language model to generate multiple possible tokenizations. During training, these are sampled using Forward Filtering Backward Sampling (FFBS), ensuring the classifier sees different segmentations of the same sentence across epochs.
To handle a dynamic vocabulary, the authors implement a Continuous Cache Embedding mechanism. This stores embeddings for recently encountered tokens; if a token is missing, it is constructed on the fly from character-level embeddings. As the classifier learns, the LM is also updated based on sampled tokens, allowing the tokenization strategy to evolve in tandem with the classification task.
Experimental Setup
The model was evaluated on several sentiment analysis and topic classification tasks across multiple languages:
- NTCIR-6: Newspaper articles in Chinese, Japanese, and English.
- Twitter Datasets: Informal social media text in Japanese and English.
- ChnSentiCorp (HOTEL): Informal Chinese reviews.
- Livedoor News: Japanese web news for topic classification.
Results and Discussion
The proposed method consistently outperformed dictionary-based and subword-based (SentencePiece) baselines across various languages and domains.
Key Findings
- Robustness through Noise: Stochastic tokenization acts as a form of data augmentation. By seeing varying segmentations of the same word, the model develops more flexible and resilient representations.
- Informal Text Performance: Unsupervised initialization (starting without a dictionary) proved superior for informal text like Twitter, where standard dictionaries often fail to capture unique slang and structures.
- Task-Awareness: Updating the LM during training allows the model to prioritize text units that are most predictive of the target label (e.g., sentiment markers).
- Cache Sensitivity: In Chinese experiments, performance was highly dependent on cache size. Increasing the cache allowed the model to match or exceed baselines despite the language's high vocabulary density.
Conclusion
Stochastic tokenization offers a powerful alternative to fixed preprocessing pipelines. By integrating tokenization into the learning process, models become more resilient to linguistic ambiguity and informal variations. Future work includes addressing the computational overhead of sampling and scaling this approach to Transformer-based architectures like BERT or GPT.
Summary updated