When we read a sentence like "The quick brown fox jumps," our brains process it not as a stream of individual letters, but as a sequence of discrete semantic units. For machines, however, identifying where one unit ends and the next begins is a fundamental challenge—especially in "unsegmented" languages like Chinese or in continuous signals like speech.
A powerful, recurring insight across decades of research is that linguistic boundaries are marked by spikes in predictive uncertainty. Within a word or a meaningful phrase, the next character or phoneme is highly predictable. But at the transition between units, the number of possible continuations explodes, causing a measurable jump in entropy.
The Intuition of Branching Entropy
Imagine you are predicting the next character in the sequence u-n-i-v-e-r-s-. Given the prefix, the next character is almost certainly i. After i, the next is likely t, and then y. Within the word "university," the "internal entropy" is low because the sequence is highly cohesive.
However, once you reach the end of the word—u-n-i-v-e-r-s-i-t-y—the next character could be a space, a comma, or the start of thousands of different words. This sudden jump in uncertainty is known as branching entropy.
Sayori Shimohata and colleagues first formalized this for collocation extraction, using boundary entropy to distinguish between complete phrases and mere fragments Retrieving Collocations by Co-occurrences and Word Order Constraints. If a string has high entropy at its boundary, it is likely a self-contained lexical unit.
Solving the Chinese Word Segmentation Gap
This principle is particularly effective for languages that do not use spaces. Traditional methods for Chinese Word Segmentation (CWS) often rely on massive, manually curated dictionaries. However, these dictionaries fail when they encounter new domains or technical jargon.
Lujun Zhao and Qi Zhang demonstrated that character-level language models can act as unsupervised segmenters Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation. By monitoring where a model's prediction confidence drops, they could identify word boundaries without explicit labels. More recently, Frank Liang showed that using these entropy peaks to "pre-tokenize" text before applying standard Byte-Pair Encoding (BPE) significantly improves how well Large Language Models (LLMs) understand unsegmented languages Entropy-Driven Pre-Tokenization for Byte-Pair Encoding.
Beyond Frequency: Information-Driven Tokenization
Most modern LLMs use BPE, which merges characters based on how frequently they appear together. However, frequency is a noisy proxy for meaning. A common sequence like t-h-e-r-e might be frequent, but it doesn't tell you if it's the word "there" or the end of "slither."
Researchers are now moving toward information-driven tokenization. Ahmed El-Kishky introduced SubwordMine, which uses entropy spikes to identify morphological boundaries like prefixes and suffixes Entropy-Based Subword Mining with an Application to Word Embeddings. By segmenting words where predictability drops, the model learns more robust representations of rare words.
Similarly, Pietro Lesci proposed ByteSpan, which replaces frequency-based merging with surprisal-based segmentation ByteSpan: Information-Driven Subword Tokenisation. By identifying "predictable spans" and cutting at "surprising" transitions, ByteSpan creates vocabularies that align much more closely with human linguistic intuition across dozens of languages.
Scaling to Speech and Semantic Alignment
The "entropy-as-boundary" signal is not limited to text. In speech processing, identifying units like syllables or words in a continuous audio stream is notoriously difficult. Alan Baade and David Harwath developed SyllableLM, which uses prediction loss spikes in a masked language model to "zoom out" from raw audio to coarse, syllable-like units SyllableLM: Learning Coarse Semantic Units for Speech Language Models. This allows the model to process speech 30x more efficiently while improving semantic understanding.
Even the process of aligning LLMs with human preferences can be optimized this way. Bolian Li and Ruqi Zhang introduced Cascade Reward Sampling (CARDS), which partitions generated text into "semantic segments" based on uncertainty spikes Cascade Reward Sampling for Efficient Decoding-Time Alignment. Instead of evaluating every single token, the system only intervenes at these natural transitions, making alignment faster and more effective.
Why It Works
The success of this approach across text, speech, and metadata Sensei: Self-Supervised Sensor Name Segmentation suggests a universal property of communication: Information is not distributed uniformly.
Meaningful units are clusters of high predictability. The "gaps" between them are the moments where the sender of a message makes a new choice, and the receiver experiences a spike in uncertainty. By measuring that uncertainty, we can reverse-engineer the structure of language itself without ever needing a dictionary.