Optimal Transport for Semantic Alignment and Representation Learning

This direction is split into other directions: semantic distribution matching. Its last version stays readable and citable here.

Version 1 of 1173 papers220 conceptsfrozen 2026-10-04

In traditional Natural Language Processing, we often measure the similarity between two pieces of text by looking for exact word overlaps or by comparing their average vector representations. However, these methods are brittle. Two sentences can share zero words yet mean the same thing, and averaging word vectors into a single "sentence embedding" often washes away the fine-grained structural nuances that define meaning.

To solve this, a growing body of research treats documents not as single points in space, but as probability distributions or "point clouds" of words. By applying Optimal Transport (OT)—a mathematical framework originally designed to find the most efficient way to move piles of dirt—researchers are developing more robust ways to align, compare, and translate human language.

The Intuition of Moving Semantic Mass

The foundational idea in this space is the Word Mover’s Distance (WMD), introduced by Matt Kusner and Kilian Q Weinberger in From Word Embeddings To Document Distances.

Imagine two documents as two different distributions of "mass" in a high-dimensional semantic space. If Document A says "The chef prepares a meal" and Document B says "A cook makes food," WMD calculates the minimum "work" required to transform the first distribution into the second. The "cost" of moving mass from "chef" to "cook" is defined by the distance between their word embeddings. Because "chef" and "cook" are close in vector space, the transport cost is low, allowing the model to recognize similarity even with zero vocabulary overlap.

Aligning Languages Without Dictionaries

One of the most powerful applications of this "mass-moving" logic is in cross-lingual learning. Traditionally, aligning two languages required massive parallel corpora or hand-crafted dictionaries. Optimal Transport allows models to align languages by looking at the global "shape" of their embedding manifolds.

Sawsan Alqahtani and colleagues at Amazon AI demonstrated in Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings that OT can serve as a fine-tuning objective. By minimizing the mathematical work required to transform a source sentence into a target sentence, the model learns semantic correspondences across languages (WordOT) without needing pre-defined translation dictionaries.

This approach is particularly vital for low-resource languages. Zhiqi Huang and James Allan developed OPTICAL, a framework that uses OT-based distillation to transfer ranking "intuition" from English to languages like Swahili or Quechua. As shown in Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation, OT provides a flexible, global mapping that accounts for differences in word order and grammar more robustly than rigid word-for-word translation.

Differentiable Alignment and the Sinkhorn Algorithm

For Optimal Transport to work within modern deep learning, it must be differentiable. Many recent papers, such as Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment by Zewen Chi and Li Dong, utilize the Sinkhorn algorithm.

The Sinkhorn algorithm adds an entropy regularization term to the transport problem, turning it into a "soft" alignment task. This produces a doubly stochastic matrix where each entry represents the probability of one token aligning with another. This "soft alignment" is used in models like STANCE (Tam et al., 2019) to detect string similarity and aliases even when characters are permuted or noisy.

Correcting Hallucinations and Improving Generation

Beyond simple distance measurement, OT is being used to "steer" generative models. In keyphrase generation, models often suffer from "hallucinations"—generating phrases unrelated to the source. Guangzhen Zhao and Guoshun Yin introduced CorrKG, which applies an OT loss to minimize the semantic discrepancy between source documents and generated sequences (Keyphrase Generation via Soft and Hard Semantic Corrections). By treating the source and the prediction as distributions, the model is penalized if the "semantic mass" of the output drifts too far from the original text.

Similarly, in simultaneous translation, Shaolei Zhang and Yang Feng use an information transport matrix to quantify exactly how much context has been "transported" from the source to the target, helping the model decide when it has enough information to start translating (Information-Transport-based Policy for Simultaneous Translation).

Interpretability and Structural Awareness

A significant advantage of OT over standard attention mechanisms is interpretability. Because OT solves for an explicit transport plan, researchers can visualize exactly which words in one sentence are being "moved" to words in another.

Dongha Lee and Hwanjo Yu leveraged this in CLRCMD, a framework for interpretable semantic textual similarity (Toward Interpretable Semantic Textual Similarity via Optimal Transport-based Contrastive Sentence Learning). Unlike black-box pooling methods, CLRCMD identifies fine-grained token alignments that mirror human judgment.

Furthermore, recent work like WSMD (Yamagiwa and Yokoi, 2023) has begun integrating structural costs into the transport calculation. By using the Fused Gromov-Wasserstein distance, these models can distinguish between sentences that share the same words but have different meanings due to word order, ensuring that the "relational shape" of the sentence is preserved during the transport process.

Papers in this version

Showing 60 of 173 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimal_transport_for_semantic_alignment_e980b268,
  title = {Optimal Transport for Semantic Alignment and Representation Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-transport-for-semantic-alignment-and-representation-learning/rdr_e980b26895154268a48f658c2e94be38},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimal_transport_for_semantic_alignment_e980b268_v1,
  title = {Optimal Transport for Semantic Alignment and Representation Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-transport-for-semantic-alignment-and-representation-learning/rdr_e980b26895154268a48f658c2e94be38/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}