In the early days of word embeddings, models like Word2Vec or fastText assigned a single, "static" vector to each word. This made cross-lingual alignment conceptually simple: if you have a point cloud of English words and a point cloud of Spanish words, you just need to find the right rotation (a linear transformation) to snap them together.
However, modern Large Language Models (LLMs) like BERT and ELMo use contextualized embeddings. In these models, a word does not have one vector; it has an infinite number of potential vectors depending on the sentence it inhabits. This "dynamic" nature creates a fundamental problem: how do you align two languages when the "points" you are trying to match are constantly shifting?
The Stability vs. Context Trade-off
Static embeddings are stable but "blind" to nuance. Contextualized embeddings are expressive but "unstable" for direct mapping. Ivan Vulić and colleagues demonstrated in Probing Pretrained Language Models for Lexical Semantics that while LLMs store rich word-level meaning, particularly in their lower layers, they lack the inherent alignment needed for cross-lingual transfer.
The research direction across these papers focuses on bridging this gap—retaining the power of context while regaining the geometric stability required for translation and reasoning.
Anchoring the Clouds
The most intuitive way to align dynamic embeddings is to turn them back into static ones temporarily. Tal Schuster and Amir Globerson pioneered an "anchoring" approach in Cross-Lingual Alignment of Contextual Word Embeddings. By averaging all the different contextual instances of a word into a single "anchor" vector, they could apply traditional linear transformations to bridge the gap between languages.
This approach works because it treats the "cloud" of a word's meanings as a single point represented by its centroid. Once the languages are aligned using these anchors, the model can then project individual, context-specific instances across the bridge.
Moving Beyond Rigid Rotations
A single rotation (isomorphism) assumes that all languages share the exact same geometric structure. But languages are not identical; English and Chinese, for instance, categorize the world differently. James Henderson and Paola Merlo argue in Multi-Adversarial Learning for Cross-Lingual Word Embeddings that we should abandon the "one-size-fits-all" transformation.
Instead of one global rotation, they use multiple adversarial networks to learn "piece-wise" mappings. This allows the model to align specific semantic subspaces differently, which is crucial for "typologically distant" languages where a single rigid transformation would fail to capture the nuances of both.
Similarly, Shengjie Luo and Di He proposed Cross-lingual Language Projection (XLP) in Revisiting Language Encoding in Learning Multilingual Representations. Rather than using a single vector to represent a language, they use projection matrices that allow the model to "re-interpret" shared representations through a language-specific lens.
Alignment as Optimal Transport
If we view a word in a language not as a point, but as a distribution of meanings, the alignment problem becomes a matter of moving "mass" from one distribution to another. Sawsan Alqahtani and Saab Mansour applied Optimal Transport (OT) to this problem in Using Optimal Transport as Alignment Objective.
By treating word alignment as a mathematical "earth mover's" problem, they can align contextualized embeddings without needing a pre-defined dictionary. This "WordOT" method allows the model to find the most efficient way to map the "cloud" of English context-vectors onto the "cloud" of German context-vectors, naturally handling the fact that one word in English might map to several different words (or senses) in German.
The Lexical Mismatch
Despite these advances, a significant hurdle remains: the gap between neural "blobs" and human "definitions." Timothee Mickus and the CODWOE team highlighted that while we can bridge different neural spaces, mapping these dynamic embeddings to the static, discrete entries in a dictionary is still incredibly difficult.
A dictionary entry for "bank" is fixed, but a BERT embedding for "bank" changes every time it appears. This mismatch between the fluid nature of modern AI and the rigid structure of human lexicography is the current frontier. Researchers like Muhao Chen are attempting to solve this by using "incidental supervision" from monolingual text to help align these fluid embeddings with structured Knowledge Graphs (Cross-lingual Entity Alignment with Incidental Supervision).
Summary of the Intuition
The shift from static to contextualized embeddings was a leap in performance but a step back for interpretability and alignment. The research direction represented here is the effort to "tame" the fluidity of contextualized models. Whether by averaging them into anchors, using piece-wise linear mappings, or applying Optimal Transport, the goal is the same: to find the stable semantic core that exists across languages, even when the mathematical representation of that core is constantly in flux.