Abstract
Cross-lingual information retrieval (CLIR) often relies on code-mixing—the synthetic blending of source and target language words—to align multilingual representations. However, this technique introduces "embedding noise" that can degrade retrieval performance, particularly in long documents. We present ContrastiveMix, a framework that disentangles language alignment from relevance matching. By applying code-mixing exclusively to queries and employing a dual-loss objective, ContrastiveMix significantly improves zero-shot cross-lingual search, especially for languages linguistically distant from English.
Introduction: The Code-Mixing Dilemma
In zero-shot CLIR, models must retrieve relevant documents in a target language (e.g., Arabic) using English queries without specific training data for that pair. Researchers typically use Multilingual Pretrained Language Models (mPLMs) and code-mixing to bridge the gap. Code-mixing replaces English words with translations (e.g., "Where is the shokudo?") to align conceptual spaces.
While effective for representation alignment, code-mixing creates a dilemma for Information Retrieval. The authors identify that mixing long passages introduces significant "embedding noise," making it harder for the model to determine precise relevance. This inconsistency between alignment needs and matching precision is the Code-Mixing Dilemma.
Methodology: Disentangled Alignment
ContrastiveMix addresses this dilemma by treating language alignment and document retrieval as distinct objectives within a dual-encoder framework.
- Dual-Loss Objective: The model optimizes two functions simultaneously:
- Relevance Loss ($L_{ir}$): A standard IR objective matching English queries to relevant English passages.
- Alignment Loss ($L_c$): A novel contrastive loss that aligns English query embeddings with their code-mixed counterparts.
- Query-Only Mixing: Unlike "NaiveMix" approaches that mix both queries and passages, ContrastiveMix applies code-mixing only to the query encoder. This is based on the observation that short queries maintain high semantic similarity (0.944 cosine similarity) when mixed, whereas long passages suffer from high variance and noise (0.867 similarity).
Experimental Results
The model was trained on the English Natural Questions (NQ) dataset and evaluated zero-shot on benchmarks like Mr.TyDi and MIRACL.
Performance on Mr.TyDi
ContrastiveMix demonstrates superior performance, particularly for "Low Similarity" languages (Arabic, Thai, Japanese, Korean, Indonesian, Finnish). As shown in the results table, ContrastiveMix consistently outperforms the standard multilingual DPR (mDPR) and NaiveMix baselines.
On average, ContrastiveMix improved MRR@100 by 1.29 points over mDPR for low-similarity languages, proving its effectiveness in bridging large linguistic gaps.
Ablation Study: The Impact of Mixing Location
To validate the "Query-Only" hypothesis, the authors compared different mixing configurations. The results confirm that mixing passages significantly degrades performance.
As seen in the table, the best performance (43.35 MRR@100) is achieved when code-mixing is applied only to the query. Mixing both components (NaiveMix) drops performance to 40.62, highlighting the noise introduced by passage-level mixing.
Discussion and Insights
The success of ContrastiveMix underscores the importance of disentangling matching from alignment. The benefits are most pronounced for non-Indo-European languages where the "alignment gap" is widest.
Limitations: The current implementation uses simple word-level replacement (MUSE dictionary), which does not account for polysemy or grammatical coherence. Future work may explore using Large Language Models (LLMs) to generate more naturalistic code-mixed queries to further refine alignment without sacrificing retrieval precision.
Summary updated