Modern natural language processing models do not read text character-by-character, nor do they process whole words at a time. Instead, they rely on tokenizers—such as Byte-Pair Encoding (BPE) or WordPiece—to break text down into intermediate units called subwords.
Traditionally, this tokenization step is entirely deterministic. Given a specific vocabulary, a text string is always sliced into the exact same sequence of tokens using a greedy, frequency-based strategy. While computationally efficient, this rigid approach introduces a hidden vulnerability: it creates brittle boundaries that obscure the internal structure of words.
Stochastic subword segmentation (often called subword regularization) fundamentally changes this paradigm. Instead of forcing a model to see a single, static token sequence during training, it introduces controlled randomness into the tokenization process. By dynamically varying how words are chopped into pieces across different training epochs, it forces neural networks to learn more resilient, flexible, and linguistically grounded representations.
The Brittle Nature of Fixed Tokenization
To understand why stochastic tokenization helps, we must first look at how deterministic tokenization fails. When a tokenizer encounters a word like unhelpful, it might consistently split it into ["un", "helpful"]. The downstream model learns to associate the combination of these two specific tokens with the word's meaning.
However, language in the real world is messy. If a user makes a simple typo, writing unhlpful, a deterministic tokenizer will completely alter its segmentation path, perhaps producing ["un", "hl", "pful"]. Because the model has only ever seen the clean, canonical split, this sudden shift in token boundaries can cause its understanding to collapse.
As demonstrated by Eric J Bigelow et al. in Forking Paths in Neural Text Generation, language models are highly sensitive to seemingly insignificant tokens like punctuation or whitespace. These minor variations act as critical decision points that can inadvertently steer a model toward incorrect outcomes. Deterministic tokenization exacerbates this fragility by turning minor character-level changes into entirely different token sequences.
Injecting Randomness into the Lattice
Stochastic subword segmentation solves this by treating tokenization not as a fixed rule, but as a probabilistic sampling problem. For any given word, there are usually multiple valid ways to decompose it using a standard vocabulary. We can visualize these choices as a tokenization lattice—a directed acyclic graph showing every possible combination of subwords that reconstructs the original text.
┌─── "unhelpful" ───┐
│ │
┌─ "un" ─┐ ┌─ "unhelp" ─┐
│ │ │ │
┌─ "u" ─┐ "helpful" "ful" "ful"
│ │
"u" "n"
During training, instead of always choosing the path with the fewest tokens or the highest frequency, the tokenizer samples paths stochastically.
Probabilistic Unigram Sampling
Taku Kudo pioneered this approach in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. By defining a subword language model, the tokenizer calculates the probability of every valid segmentation path in the lattice. During training, it samples a path based on these probabilities, exposing the model to a rich variety of word decompositions.
BPE-Dropout
While effective, sampling from a full probability lattice can add computational overhead. To simplify this, Ivan Provilkov et al. introduced a streamlined alternative in BPE-Dropout: Simple and Effective Subword Regularization. Instead of changing the underlying tokenizer framework, BPE-Dropout takes a standard BPE merge table and randomly hides or "drops" merge operations with a fixed probability during training. If the rule to merge un and helpful is dropped, the tokenizer is forced to output them as separate fragments, naturally producing varied segmentations on the fly.
Stochastic Token Expansion
Rather than altering the tokenization boundaries of the input text directly, another approach involves expanding standard tokens into smaller sub-units. In StochasTok: Improving Fine-Grained Subword Understanding in LLMs, Anya Sims et al. proposed randomly decomposing standard tokens into smaller, equivalent sub-units already present in the vocabulary. This allows the model to observe the internal structure of words without requiring any modifications to the underlying vocabulary size.
Why It Works: Building Intuition
Why does adding noise to the input representation make a model perform better? The intuition relies on three core principles: segmentation-agnostic representations, entropy maximization, and morphological awareness.
1. Segmentation-Agnostic Representations
When a model is trained exclusively on a single deterministic tokenization, it overfits to those specific token boundaries. By forcing the model to process a word like strawberry sometimes as ["strawberry"], sometimes as ["straw", "berry"], and sometimes as ["s", "t", "r", "a", "w", "b", "e", "r", "r", "y"], the model learns that all of these distinct sequences map to the exact same semantic concept.
Consequently, if a typo or character perturbation shifts the token boundaries at runtime, the model does not panic. It has already learned to compose meaning from subwords of varying granularities. As shown by Yekun Chai et al. in Tokenization Falling Short: On Subword Robustness in Large Language Models, this stochastic fragmentation acts as a "vaccine" that significantly improves a model's awareness of internal character structures and its resilience to typographical errors.
2. Maximizing Tokenization Entropy
Standard frequency-based tokenizers suffer from a heavy bias toward a tiny subset of highly frequent token combinations. In Distributional Properties of Subword Regularization, Marco Cognetta et al. demonstrated that maximizing the entropy of the subword distribution—such as through uniform sampling across the entire space of valid decompositions—acts as a powerful form of data augmentation. It prevents the model from collapsing onto brittle corpus statistics and forces it to process less frequent subword sequences, leading to more generalized linguistic features.
3. Morphological Awareness
Fixed tokenization boundaries often fail to align with actual linguistic units, which is particularly problematic for languages with complex or non-concatenative morphology (where words change internally rather than just adding prefixes or suffixes). Chantal Amrhein et al. argued in How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology? that introducing stochasticity into subword segmentation forces the model to learn from multiple possible variations of the same root word. This prevents over-reliance on fixed boundaries and allows the model to better generalize across complex internal word structures.
Balancing Training Noise with Inference Efficiency
One of the most elegant properties of stochastic subword segmentation is that the randomness is typically restricted to the training phase.
During training, the model acts as an implicit ensemble, learning to process text across an array of different token granularities. But during inference, the randomness can be turned off entirely. The tokenizer reverts to its stable, deterministic canonical paths, ensuring that decoding remains fast, predictable, and computationally efficient.
As explored by Sho Takase et al. in Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation, this asymmetry allows downstream models to fully retain the regularization benefits of training-time diversity while maintaining the speed and simplicity of deterministic segmentation at runtime. By changing how the model sees text during learning, we build systems that truly understand the compositionality of language rather than just memorizing the arbitrary choices of a tokenizer.