Atomic Representations of Multiword Phrases and Entities

This direction is merged into another direction: atomic phrase representation. Its last version stays readable and citable here.

Version 1 of 1112 papers139 conceptsfrozen 2026-10-04

In standard Natural Language Processing, we typically treat sentences as sequences of individual tokens. We assume that the meaning of a sentence can be built up by composing these tokens—adding or multiplying their vectors together. However, language is rarely so neatly additive. Phrases like "White House," "not bad," or "chronic obstructive pulmonary disease" possess a unified semantic identity that is often lost when they are decomposed into their constituent parts.

A significant body of research argues that to truly understand language, models must move beyond word-level processing and treat multiword expressions (MWEs) and entities as atomic semantic units.

The Failure of Compositionality

The primary motivation for this shift is the "compositionality trap." Standard models assume that the vector for "hot dog" should be some combination of "hot" and "dog." But a hot dog is neither a temperature nor a canine. When models treat these as independent units, they suffer from semantic drift.

Wenpeng Yin and Hinrich Schuetze demonstrated in An Exploration of Embeddings for Generalized Phrases that assigning a dedicated vector to an entire multiword unit captures non-compositional meanings that additive models miss. This is particularly vital for sentiment analysis. As Duyu Tang and colleagues showed in A Joint Segmentation and Classification Framework for Sentiment Analysis, treating "not bad" as an atomic unit prevents "polarity inconsistency," where the negative sentiment of "not" and the neutral sentiment of "bad" might otherwise confuse a classifier.

Holistic Embedding Strategies

How do we represent these units? The most direct approach is to modify the input text before training. By identifying collocations and replacing them with a single joint token (e.g., white_house), researchers can use standard algorithms like Skip-gram to learn a unique position in semantic space for that phrase.

This technique has proven essential in specialized domains:

  • Medicine: Yu Hao and Ji Wu found that word-level models struggle with complex diagnoses. Their CAMSE framework focuses on capturing terminologies that span multiple tokens to maintain clinical meaning.
  • Construction: Antoine J.‐P. Tixier showed that domain-specific "small data" can outperform generic "big data" if the model is trained to recognize industry-specific multiword phrases as single units.
  • Chemistry: In the study of energetic materials, Daniel C. Elton and his team found that removing spaces and dashes to create single-token representations of chemical names was necessary to prevent the loss of specific technical meanings.

From Tokens to Spans

While pre-tokenization works for fixed phrases, it is less flexible for dynamic entities. A more recent evolution is span-level modeling. Instead of tagging each word in "New York City" as part of a location, span-based models treat the entire sequence as a single candidate for classification.

Shuo Liang and Wei Wei developed the STAGE framework to extract sentiment triplets by treating every possible word sequence as a candidate span. This avoids the fragmentation common in word-level sequence labeling. Similarly, Lu Xu and Yew Ken Chia showed in Learning Span-Level Interactions for Aspect Sentiment Triplet Extraction that modeling the relationship between entire spans—rather than word-to-word interactions—significantly improves the identification of complex targets.

Preserving Integrity in Knowledge Graphs

The need for atomic representation extends to how AI interacts with structured data. In knowledge graphs, entities are often multiword names. Bishan Yang and Wen-tau Yih found that representing these entities as single semantic units using pre-trained phrase embeddings significantly improves link prediction accuracy.

This concept also bridges the gap between different domains. Carlotta Domeniconi and Pu Wang used Wikipedia concepts as a "semantic bridge" for cross-domain classification. By mapping documents to multiword Wikipedia entities rather than raw words, they enabled models to transfer knowledge between domains with entirely different vocabularies.

Why It Works: The Intuition of Cohesion

The core intuition behind this research is semantic integrity. When we break a phrase like "Hidden Markov Model" into three parts, we are left with a word about concealment, a surname, and a generic term for a representation. None of these individual parts contain the mathematical concept of the whole.

By collapsing these sequences into atomic units—whether through re-tokenization (as in Extracting Topical Phrases from Clinical Documents by Yulan He), span-level attention (as in GEI by Gang Zhao), or joint embeddings (as in HCE by Zhiting Hu)—we allow the model to learn the context of the concept rather than the context of the words. This shift from lexical to conceptual processing is what enables AI to handle the nuance of human language, from technical jargon to idiomatic expressions.

Papers in this version

Showing 60 of 112 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_atomic_representations_of_multiword_phra_9a2eb5b4,
  title = {Atomic Representations of Multiword Phrases and Entities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/atomic-representations-of-multiword-phrases-and-entities/rdr_9a2eb5b4965044ca969022898689bffc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_atomic_representations_of_multiword_phra_9a2eb5b4_v1,
  title = {Atomic Representations of Multiword Phrases and Entities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/atomic-representations-of-multiword-phrases-and-entities/rdr_9a2eb5b4965044ca969022898689bffc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}