When a human reads the phrase "hot dog," they do not think of a high-temperature canine. Instead, they instantly retrieve a single, unified concept: a sausage in a sliced bun.
However, standard language models process text through subword tokenizers (such as Byte-Pair Encoding or WordPiece) or unigram bag-of-words representations. These approaches break text down into individual fragments and rely on subsequent layers to compose their meaning. While this compositional assumption works perfectly for phrases like "red car," it fails for non-compositional language. Idioms ("kick the bucket"), technical terms ("hidden Markov model"), and domain-specific entities ("white blood cell") lose their specialized meaning when decomposed into generic components.
To solve this, a prominent research direction focuses on atomic phrase representation: identifying multiword expressions (MWEs) and technical entities before tokenization and treating them as single, indivisible units within a model's vocabulary.
Overcoming the Compositional Bias
The primary intuition behind atomic phrase representation is that certain word combinations possess a holistic semantic identity that cannot be reliably reconstructed from their parts.
Subword tokenizers naturally bias models toward literal interpretations. For example, Wei He 0017 et al. point out that when a model applies standard compositional logic to figurative phrases, it dilutes their actual meaning. By treating idiomatic expressions as atomic units during preprocessing, models are prevented from incorrectly breaking them down (Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss). This strategy is equally vital across languages. Jing Jiang et al. demonstrate that representing non-compositional Chinese idioms (Chengyu) as unique embedding vectors preserves metaphorical meanings that literal character components fail to convey (Learning and Evaluating Chinese Idiom Embeddings).
Standard Tokenization: [white] + [blood] + [cell] --> Compositional drift
Atomic Tokenization: [white_blood_cell] --> Preserved semantic identity
This preservation is also critical for cross-lingual alignment. A multiword phrase in one language often maps to a single word in another (e.g., "hot dog" in English to "pancho" or "perrito caliente"). Naoki Otani et al. find that pre-tokenizing MWEs as single units allows models to accurately map complex phrases to single-word equivalents in other languages. Crucially, this isolation prevents individual words from inheriting the distributional properties of the larger expressions they inhabit, protecting the independent meanings of words like "hot" and "dog" (Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings).
Structural and Computational Advantages
Beyond preserving semantics, grouping words into atomic tokens fundamentally alters how models navigate search spaces and syntactic structures.
Reducing Generation Search Space
In sequence generation and machine translation, predicting text word-by-word introduces a massive combinatorial search space. If a model treats an idiom or a named entity as a sequence of independent subwords, it risks generating unnatural word permutations.
The authors of Handling Multiword Expressions in Phrase-Based Statistical Machine Translation address this by concatenating MWE components during preprocessing, forcing alignment models to treat them as atomic units. Similarly, Andrea Zaninello et al. show that token-level concatenation prevents neural machine translation models from decomposing idiomatic phrases, leading to substantial gains in translation quality (Multiword Expression aware Neural Machine Translation). For open-ended generation, Deirdre Hogan et al. argue that treating MWEs as single tokens simplifies surface realization, reducing the search space and ensuring that fixed phrases maintain their internal structure (Exploiting Multi-Word Units in History-Based Probabilistic Generation).
Eliminating Syntactic Noise
When parsing sentences, fixed expressions often exhibit irregular internal grammar that can confuse a model. For instance, the phrase "by and large" consists of a preposition, a conjunction, and an adjective, yet it functions collectively as an adverb.
Matthieu Constant et al. demonstrate that representing fixed MWEs as atomic nodes within a dependency tree reduces structural noise. By bypassing the irregular internal grammar of the phrase, the parser can focus entirely on the broader structural role of the expression within the sentence, leading to higher parsing accuracy (A Transition-Based System for Joint Lexical and Syntactic Analysis).
How Phrases Are Bound
To implement atomic representations, pipelines must first identify which words belong together. Researchers rely on two primary mechanisms: statistical phrase mining and external knowledge integration.
- Statistical Phrase Mining: This data-driven approach uses corpus statistics to find highly correlated contiguous words. For instance, Seán Matthews et al. utilize Normalized Point-wise Mutual Information (NPMI) to identify and group domain-specific jargon in legal opinions before training embeddings, ensuring that complex legal terms are not fractured (Gender and Racial Stereotype Detection in Legal Opinion Word Embeddings).
- Knowledge Base Integration: Other frameworks map text directly to structured resources. Walid Shalaby et al. use external knowledge graphs and Wikipedia links to isolate multi-word entity mentions, treating them as atomic tokens during training to prevent vector averaging from diluting their distinct semantic identity (Beyond Word Embeddings: Learning Entity and Concept Representations from Large Scale Knowledge Bases).
The Trade-off: Compositional Loss
While atomic phrase representation prevents semantic dilution, it introduces a distinct trade-off: the complete loss of internal compositionality.
If every technical phrase or entity is turned into an indivisible token, the model can no longer exploit shared sub-components to understand rare or unseen variations. Simon Suster notes this limitation in medical reading comprehension, warning that treating complex multi-word technical terms as purely atomic tokens strips away internal semantic information and compositional meaning (CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension).
Furthermore, this atomicity can hurt information retrieval flexibility. As Robert Krovetz observed, indexing multi-word phrases strictly as single units can decrease search effectiveness because it eliminates partial credit; a document containing only a portion of a phrase becomes undiscoverable unless the exact, unbroken atomic unit is matched (Homonymy and Polysemy in Information Retrieval).
The design choice between atomic and compositional representations therefore hinges on the nature of the text: for highly literal language, compositionality reigns; for idioms, specialized jargon, and rigid entities, atomicity is essential.
Key Foundational Work
Tokenization as the Initial Phase in NLP (1992) — contributed 2 concept elements to this direction.