Optimal Transport for Semantic Text Alignment

113 concepts79 papers1 proposal

When comparing two pieces of text, traditional natural language processing metrics face a difficult trade-off. Simple lexical overlap metrics like BLEU or ROUGE are brittle; they fail to recognize synonyms or paraphrases. Conversely, modern vector-space approaches often collapse an entire document into a single averaged embedding vector. While vector averaging captures a broad semantic gist, it washes out the internal geometric structure of the text. If a document discusses "dogs" and "cats," averaging their vectors yields a point in the middle that may represent an entirely different concept.

To resolve this, a powerful paradigm treats documents not as single static vectors, but as probability distributions over an embedding space. By leveraging the mathematical framework of Optimal Transport (OT), we can calculate the distance between these distributions. This approach allows for soft, token-level alignment that preserves the geometric relationships between individual words.

Text as a Probability Distribution

Imagine a sentence as a point cloud in a high-dimensional embedding space. Each unique word in the sentence represents a point, and we can assign a weight (or probability mass) to that point proportional to its frequency.

If we want to measure the distance between two sentences, we can frame it as an optimization problem: what is the minimum computational effort required to transform one point cloud into the other?

Document A: "The physician treats patients."
               │             │
               ▼ (Low Cost)  ▼ (Low Cost)
               ▼             ▼
Document B: "The doctor heals people."

In the language of optimal transport, this is known as the Earth Mover's Distance (EMD). The "earth" is the probability mass of the words in Document A, and we want to move it into the "holes" represented by the words in Document B.

The Mechanics of Word Mover's Distance

In their seminal paper From Word Embeddings To Document Distances, Matt Kusner et al. formalized this concept for natural language as the Word Mover's Distance (WMD).

The elegance of WMD lies in how it constructs its cost matrix. The cost $c(i, j)$ of transporting a unit of mass from word $i$ in Document A to word $j$ in Document B is explicitly defined as the Euclidean distance between their respective word embeddings:

$$c(i, j) = \|x_i - x_j\|_2$$

The algorithm then solves for a transport plan $T_{i,j}$—which dictates how much mass to move from word $i$ to word $j$—that minimizes the total cumulative cost:

$$\min_{T \ge 0} \sum_{i,j} T_{i,j} c(i, j)$$

Subject to the constraints that all mass from Document A must be fully distributed to match the mass of Document B.

Why This Built-In Intuition Works

Because the underlying cost matrix relies on semantic distance, the optimal transport plan naturally finds the most efficient semantic alignment. If Document A contains the word "physician" and Document B contains "doctor", the distance between their embeddings is small. The optimization framework will naturally route mass between them at a very low cost. If Document B instead contained an unrelated word like "apple", the transport cost would be high.

As a result, WMD can robustly identify semantic similarity even when two documents share zero vocabulary words in common.

From Static to Contextualized Alignments

While static word embeddings allowed early optimal transport metrics to capture synonymy, they struggled with polysemy—where a single word's meaning shifts based on its surroundings. To bridge this gap, researchers shifted from static point clouds to contextualized distributions.

Wei Zhao proposed MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance, which swaps out static vectors for contextualized representations derived from deep language models like BERT. By calculating the Earth Mover's Distance over contextualized distributions, the metric can perform soft token alignment that respects the exact nuance of how a word is used in a specific sentence.

This distributional view scales hierarchically. For instance, when evaluating multi-sentence documents, treating a text as a simple bag-of-words discards narrative structure. To preserve this grouping, Elizabeth Clark et al. introduced Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. This variant treats a document as a distribution of sentence-level embeddings, calculating the minimum cost to transport entire sentences and effectively mapping the structural flow of ideas from one text to another.

Optimal Transport as a Training Objective

If optimal transport provides a robust, human-aligned measure of semantic distance, a natural next step is to use it directly as a loss function to train generative models.

Standard autoregressive language models are trained using cross-entropy loss, which operates on discrete token matches. Cross-entropy penalizes all incorrect tokens equally; if the target word is "astronaut", predicting "cosmonaut" incurs the exact same penalty as predicting "banana". This lack of granularity causes exposure bias and limits semantic flexibility.

To inject semantic awareness into sequence-to-sequence models, Liqun Chen et al. proposed an optimal-transport-based sequence loss in Improving Sequence-to-Sequence Learning via Optimal Transport. By minimizing the transport cost between the generated token distribution and the target distribution during training, the model receives a continuous, differentiable signal that rewards it for generating semantically equivalent phrases, even if they don't match the reference text word-for-word.

Building on this, Siyu Ren et al. developed EMO: EARTH MOVER DISTANCE OPTIMIZATION FOR AUTO-REGRESSIVE LANGUAGE MODELING. EMO computes a semantic transport cost using the cosine distance between token embeddings in the model's output layer.

Cross-Entropy Loss:
Predicted: "cosmonaut" ──► Wrong (Full Penalty)
Predicted: "banana"     ──► Wrong (Full Penalty)

Earth Mover Distance Optimization (EMO):
Predicted: "cosmonaut" ──► Close in embedding space ──► Low Penalty
Predicted: "banana"     ──► Far in embedding space   ──► High Penalty

By optimizing a differentiable upper bound of the Earth Mover's Distance, the model learns to distribute its probability mass to semantically related words, preventing catastrophic generation failures when it deviates from the exact training tokens.

Structuring Latent Spaces in Topic Modeling

The benefits of geometry-aware distribution matching extend beyond text generation into unsupervised text clustering. Traditional topic modeling frameworks, such as Latent Dirichlet Allocation (LDA), rely strictly on discrete word co-occurrence statistics. Consequently, they often fail on short, sparse texts like tweets or product descriptions where word overlap is minimal.

By reframing topic modeling through the lens of optimal transport, Viet Huynh et al. introduced OTLDA: A Geometry-aware Optimal Transport Approach for Topic Modeling. Instead of treating documents and topics as discrete probability vectors over a vocabulary index, the authors model them as distributions in a continuous embedding space.

By minimizing the Wasserstein distance between document word distributions and latent topic distributions, the model leverages the pre-trained geometry of word embeddings. This geometric alignment ensures that words with similar meanings are mapped to the same latent topics, preventing topic collapse and allowing the model to discover coherent themes in highly sparse text environments.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_distribution_matching_02bada9f,
  title = {semantic distribution matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-distribution-matching/rdr_02bada9f1ad2493099052bd1bcd51c38},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_distribution_matching_02bada9f_v1,
  title = {semantic distribution matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-distribution-matching/rdr_02bada9f1ad2493099052bd1bcd51c38/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake