Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint

Dayeon Ki, Cheonbok Park, Hyunjoong Kim

Introduction

Current cross-lingual sentence embeddings often suffer from a phenomenon termed "semantic leakage," where language-specific information bleeds into semantic representations. This hinders effective data mining and cross-lingual retrieval. This paper introduces ORACLE (ORthogonality Constraint LEarning), a novel training objective that enforces strict orthogonality between semantic and language-specific embeddings. By ensuring these two spaces are mathematically independent, ORACLE significantly improves semantic alignment and retrieval accuracy across diverse language pairs.

The Challenge of Semantic Leakage

To build effective Neural Machine Translation (NMT) systems, researchers require massive amounts of parallel data. Since high-quality parallel text is scarce for many languages, it is often "mined" from the web using cross-lingual sentence embeddings. The goal is to create a shared embedding space where sentences with the same meaning cluster together, regardless of language.

To achieve this, models attempt to disentangle an embedding into two parts: a "semantic" component (the meaning) and a "language-specific" component (syntax and idioms). However, the authors identify a critical flaw: current disentanglement methods fail to fully separate these components. Language-specific information unintentionally leaks into the semantic representation, causing models to cluster sentences by language rather than by meaning. This significantly degrades the accuracy of parallel pair retrieval.

The ORACLE Framework

The authors propose ORACLE, a multi-task training objective designed to be implemented atop existing disentanglement frameworks like DREAM or MEAT. ORACLE is "plug-and-play," meaning it can refine the output of various pre-trained multilingual encoders, such as LASER, InfoXLM, or LaBSE.

The ORACLE architecture featuring intra-class clustering and inter-class separation.

As illustrated in Figure 3, ORACLE consists of two primary components:

  1. Intra-class Clustering ($L_{IC}$): This forces language-specific embeddings of the same language to cluster closely together. By tightening these clusters, the model is discouraged from spreading language-specific "noise" into the semantic space.
  2. Inter-class Separation ($L_{IS}$): This is the core innovation. It enforces an orthogonality constraint—mathematically ensuring that the semantic vector and the language vector for a given sentence are at a 90-degree angle (orthogonal) to one another. This minimizes their cosine similarity to zero, rendering the two representations independent.

Experimental Evaluation

The researchers compiled a massive training dataset of 6 million parallel sentences covering 12 language pairs, ranging from high-resource German to low-resource Aymara and Guaraní. Evaluation was conducted using standard benchmarks including Tatoeba-14 and Tatoeba-36 for cross-lingual sentence retrieval, and SemEval-2017 for Semantic Textual Similarity (STS) tasks.

Key Results and Insights

The integration of ORACLE led to significant performance gains across all tested encoders. On the Tatoeba-36 benchmark, semantic retrieval accuracy reached state-of-the-art levels, with LaBSE improving to an average of 96.36%.

Spearman's rank correlation for semantic and language tasks across different encoders with the addition of ORACLE.

As shown in Figure 5, ORACLE consistently improved Spearman’s rank correlation on STS tasks. Simultaneously, "language leakage"—measured by the ability of language embeddings to perform semantic tasks—dropped drastically. For LaBSE, this leakage fell from 16.92% to 1.96%, indicating much cleaner disentanglement.

Furthermore, t-SNE visualizations provided qualitative evidence of this separation. In vanilla models, semantic and language clusters often overlap in a messy cloud. With ORACLE, language-specific clusters move to the periphery, leaving a clean, language-agnostic semantic core. The model also showed increased robustness to code-switching, as it learned to ignore language-specific tokens when calculating semantic vectors.

Conclusion

ORACLE provides a robust, mathematically grounded method for mitigating semantic leakage in cross-lingual embeddings. While the study noted a slight trade-off in alignment for specific pairs, the overall improvement in retrieval and semantic similarity tasks is substantial. Future work will focus on expanding the language scope and optimizing the balance between separation and alignment.

Summary updated

BibTeX
@inproceedings{ki2024mitigating,
  author = {Dayeon Ki and Cheonbok Park and Hyunjoong Kim},
  title = {{Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure