Deep learning models cannot process raw text directly; they require discrete numerical inputs. Historically, natural language processing faced a harsh dichotomy: model at the word level or the character level. Word-level models capture rich semantics but suffer from the out-of-vocabulary (OOV) problem and require massive, computationally expensive vocabularies. Character-level models eliminate OOV errors entirely, but result in long sequence lengths that dilute semantic density and strain model attention spans.
Subword tokenization bridges this gap by statistically decomposing text into frequent character sequences. As noted by Moussa Kamal Eddine et al. in BARThez: a Skilled Pretrained French Sequence-to-Sequence Model, this approach balances the fine-grained flexibility of character-level models with the semantic richness of word-level models. By doing so, it provides a flexible framework for handling large vocabularies and morphological complexity across diverse languages.
The Statistical Engine of Subword Discovery
Instead of relying on manual linguistic rules, subword tokenization algorithms use data-driven statistical optimization to determine where to split a word. The most prominent methods include Byte Pair Encoding (BPE), WordPiece, and Unigram segmentation.
BPE begins by treating every character as an individual token and iteratively merges the most frequently co-occurring token pairs until a predefined vocabulary size is reached. This iterative merging logic can even be applied to other domains, such as sign language translation, where Taro Miyazaki et al. introduce Gloss Pair Encoding in Sign Language Translation with Gloss Pair Encoding to merge co-occurring sign glosses into single, more meaningful tokens.
The choice of algorithm and vocabulary size heavily influences downstream performance. For instance, Kurt Abela et al. systematically compared word-level and subword techniques in Tokenisation in Machine Translation Does Matter: The impact of different tokenisation approaches for Maltese, finding that BPE with an 8,000 vocabulary size provided the optimal configuration for morphologically rich languages. Similarly, in Benchmarking Azerbaijani Neural Machine Translation, Chih-Chen Chen et al. demonstrated that Unigram segmentation can outperform BPE by optimizing a probabilistic language model to select subwords based on their statistical likelihood.
By using statistical likelihoods, models can achieve variable-length tokenization. As shown in Variable-Length Word Encodings for Neural Translation Models, techniques like Huffman coding allow a large vocabulary to be mapped onto a small, fixed set of tokens. This ensures that rare words are represented as sequences of frequent symbols, completely eliminating the need for an information-destroying unknown token (<UNK>). Once these subwords are generated, marker-based subword detokenization—as discussed by Mohammed Bennamoun et al. in SubICap: Towards Subword-informed Image Captioning—uses boundary indicators to seamlessly stitch the subwords back into human-readable text.
Capturing Morphology and Domain Jargon
Why does subword tokenization work so well across diverse languages and technical fields? The answer lies in its ability to implicitly learn morphology—the internal structure of words. In morphologically rich languages, words change form constantly via prefixes and suffixes. Rather than memorizing every variation, subword tokenization isolates the root and the affixes.
Prashant Mathur et al. leverage this property in Multi-lingual neural title generation for e-Commerce browse pages, demonstrating that BPE allows models to handle complex morphological inflections and rare words without relying on rigid placeholder systems. This morphological awareness is also crucial for low-resource languages; Ife Adebara et al. show in AfroLID: A Neural Language Identification Tool for African Languages that BPE balances unique roots with orthographic variations, outperforming character-level models.
This capability extends directly to specialized domains filled with complex jargon. In biological and medical text, new or rare terms are frequently constructed from familiar Latin or Greek roots. Emma Strubell et al. demonstrate in Attending to All Mention Pairs for Full Abstract Biological Relation Extraction that subword tokenization allows models to interpret unseen technical terms through their constituent parts. Similarly, Benjamin Hou et al. find in RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting that decomposing complex medical terms into sub-units mitigates out-of-vocabulary errors when describing rare clinical conditions.
Even in highly structured domains like programming languages or network security, subwords provide a compact representation. Hieu Tran et al. show in SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs that standard natural language subword vocabularies can successfully decompose complex URIs and camelCase strings in SPARQL queries, avoiding the need for massive entity-specific vocabularies. In cybersecurity, Abdullah Bin Jasni et al. apply subword tokenization to hexadecimal sequences in DiffuPac: Contextual Mimicry in Adversarial Packets Generation via Diffusion Model, treating raw network packet data as a structured language.
Granular Encodings: Bytes and Sub-Characters
While standard subword tokenization operates on characters, it struggles with non-standard text, typos, or historical documents where spelling is highly irregular. To solve this, researchers shifted the granularity down to the byte level.
Byte-level Byte Pair Encoding (BBPE) represents text as a sequence of UTF-8 bytes rather than characters. As Abbas Ghaddar et al. explain in Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing, BBPE ensures that any string can be decomposed into a valid sequence of tokens, completely eliminating out-of-vocabulary terms. This is particularly powerful for historical texts; Vallari Pai demonstrates in Automatic Transcription of Handwritten Old Occitan Language that byte-level BPE processes text effectively without requiring a standardized vocabulary. The authors of From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French also leverage byte-level subword encoding to handle erratic spelling variations and archaic symbols.
However, standard byte encodings can lead to over-segmentation in non-Latin scripts, where a single character might require multiple bytes, resulting in unnecessarily long sequences. To address this inequity, Tomasz Limisiewicz et al. developed MYTE in MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling. By mapping the smallest meaningful units of language (morphemes) directly to compact byte sequences, MYTE reduces sequence lengths and computational overhead for non-Latin scripts.
For languages with complex syllabic blocks, such as Korean, tokenization can go even deeper into sub-character components. Jean Lee et al. demonstrate in K-MHaS: A Multi-label Hate Speech Detection Dataset in Korean Online News Comment that decomposing complex characters into their fundamental phonetic or structural components makes models highly robust against intentional character modifications, slang, and obfuscated text.
Structural Artifacts and Tokenization Flaws
Despite its success, subword tokenization introduces subtle structural flaws because the statistical optimization of a vocabulary does not perfectly align with human semantics.
One major issue is the creation of "glitch tokens." As Sander Land et al. reveal in Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models, BPE algorithms often generate intermediate fragments or character sequences that are never used independently in the training corpus. These artifacts occupy space in the vocabulary but lack meaningful representations in the learned embedding space, leading to unpredictable model behavior.
Furthermore, forcing a model to exhaustively segment every character can introduce noise. To combat this, Valentin Hofmann et al. proposed FLOTA in An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers. FLOTA uses non-exhaustive subword segmentation, leaving certain characters untokenized if they do not form a valid vocabulary entry within a set iteration limit, thereby avoiding the noise of meaningless fragments. To mathematically reconcile these token-level abstractions with raw text, Tim Vieira et al. introduced a framework in From Language Models over Tokens to Language Models over Characters that uses a minimal token covering to map token-level models back to character-level probabilities, solving boundary alignment issues.
Cross-Lingual Transfer and Cross-Modal Adaptation
Because subword tokenization patterns naturally reflect a language's underlying morphology, they serve as a powerful signal for cross-lingual transfer. Tanja Samardżić et al. demonstrated in On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers that BPE subword tokenization patterns can be used as a metric to predict the success of cross-lingual parsing transfer between related languages. This shared vocabulary space is also exploited by Noe Casas et al. in The TALP-UPC Machine Translation Systems for WMT18 News Shared Translation Task, where wordpiece extraction allows low-resource languages to share vocabulary and capture morphological similarities.
Remarkably, the intuition behind subword tokenization transfers seamlessly to audio and speech processing. Instead of tokenizing text characters, models can tokenize sequences of discovered acoustic units or phonemes. Ziyang Ma et al. explore this in Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation, applying BPE to sequences of discovered phonemes to identify and group recurring sound patterns into a hierarchical vocabulary. This phonetic subword tokenization is highly effective for polysynthetic or endangered languages where complex word structures make word-level modeling impractical, as demonstrated by Gilles Boulianne in Phoneme transcription of endangered languages: an evaluation of recent ASR architectures in the single speaker scenario.