Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint

Dayeon Ki, Cheonbok Park, Hyunjoong Kim

TL;DR

This paper tackles "semantic leakage" in cross-lingual sentence embeddings, where language-specific information unintentionally contaminates semantic representations. They introduce ORACLE, a novel training objective that enforces orthogonality between semantic and language embeddings, leading to improved cross-lingual alignment and reduced leakage.

What Problem Does This Paper Address?

Accurately aligning sentences with the same meaning across different languages is crucial for tasks like cross-lingual information retrieval and machine translation, especially for low-resource languages. Multilingual pre-trained models (like mBERT and XLM-R) are often used to create unified embedding spaces. However, a common issue is "semantic leakage," where the embeddings for a sentence inadvertently contain information about its language, rather than purely its meaning. This makes it hard to truly disentangle semantics from language, hindering the effectiveness of these embeddings. Previous work has focused on aligning semantics but largely overlooked the separation of language-specific information.

Methods

The core of this paper is the ORACLE (ORthogonality Constraint LEarning) objective. ORACLE is designed to be plugged into existing disentanglement methods. It comprises two key components:

  • Intra-class Clustering (LIC): This component aims to pull together language-specific representations of the same language. The authors observed that previous methods lacked this explicit constraint, leading to language embeddings being scattered. By minimizing the pairwise cosine distance between language embeddings of the same language, ORACLE encourages them to cluster.
  • Inter-class Separation (LIS): This component enforces orthogonality between different types of representations. Specifically, it aims to make semantic representations orthogonal to language-specific representations. This is achieved by minimizing a term that encourages the cosine similarity between semantic and language embeddings to be as close to zero as possible.

ORACLE is trained by optimizing a combined loss function that includes LIC and LIS. The authors integrate ORACLE with existing disentanglement frameworks like DREAM and MEAT, using separate Multi-Layer Perceptrons (MLPs) to extract semantic and language representations from initial sentence embeddings.

Datasets

The paper uses publicly available bilingual corpora to create a dataset comprising 12 language pairs, with English as the source language. They randomly sample 0.5 million sentences for each language pair, splitting them into 0.45M for training and 0.05M for testing, totaling 6 million parallel sentences.

They evaluate their method on:

  • Held-out Test Set: A custom dataset created from the bilingual corpora.
  • Tatoeba: A public dataset for cross-lingual sentence retrieval.
  • SemEval-2017 Semantic Textual Similarity (STS) task: A standard benchmark for evaluating semantic similarity.
  • Code-switched dataset: Manually created by replacing words in source sentences with their translations from bilingual dictionaries (sourced from MUSE).

Key Insights and Ideas

  • Semantic Leakage is a Real Problem: The paper clearly defines and demonstrates "semantic leakage" as a significant issue that hinders effective cross-lingual alignment.
  • Orthogonality is Key for Separation: The core idea that enforcing orthogonality between semantic and language representations is crucial for effective disentanglement is a key insight.
  • Intra-class Clustering Matters: The authors found that explicitly clustering language embeddings within themselves (LIC) is important, not just separating semantics from language.
  • ORACLE is Complementary: ORACLE works best when integrated with existing methods like DREAM and MEAT, suggesting it's a powerful add-on rather than a complete replacement.
  • Improved Performance on Unseen Languages: ORACLE shows a greater impact on improving semantic alignment for unseen languages compared to seen languages, suggesting it helps bridge the gap.

Key Results

  • Cross-lingual Sentence Retrieval: Applying ORACLE consistently shifts performance towards higher semantic retrieval accuracy and lower language embedding retrieval accuracy across various multilingual encoders (LASER, InfoXLM, LaBSE) and baselines (DREAM, MEAT). For instance, on the held-out test set, ORACLE generally leads to better semantic accuracy and reduced language accuracy. On the Tatoeba dataset, ORACLE-enhanced models generally outperform existing methods.
  • Semantic Textual Similarity (STS): ORACLE leads to a stronger positive correlation with STS scores for semantic representations and a stronger negative correlation for language representations, indicating better semantic alignment and effective separation, respectively.
  • Code-Switching: ORACLE enhances both semantic and language embedding accuracy even in challenging code-switched scenarios, demonstrating its robustness.
  • Visualization: Visualizations of embedding spaces clearly show that ORACLE mitigates semantic leakage, leading to clearer separation of language-specific representations while maintaining semantic alignment.

Limitations and Future Work

  • Integration Dependency: While ORACLE can be used as a stand-alone objective, it yields better results when integrated with existing methods like DREAM or MEAT. This limits its direct application and opens avenues for exploring optimal combinations.
  • Semantic Alignment Trade-off: For certain language pairs, there's a slight trade-off in semantic alignment when addressing semantic leakage. Future work could explore methods that mitigate leakage without compromising semantic quality.
  • Limited Language Pairs: The experiments were conducted on 12 language pairs. Expanding the study to a wider array of languages and multilingual encoders is suggested for future work.

Summary updated

BibTeX
@inproceedings{ki2024mitigating,
  author = {Dayeon Ki and Cheonbok Park and Hyunjoong Kim},
  title = {{Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure