Abstract
Despite advances in the multilingual capabilities of Large Language Models (LLMs), their performance varies substantially across different languages and tasks. In multilingual retrieval-augmented generation (RAG)-based systems, knowledge bases (KB) are often shared from high-resource languages (such as English) to low-resource ones, resulting in retrieved information from the KB being in a different language than the rest of the context. In such scenarios, two common practices are pre-translation to create a mono-lingual prompt and cross-lingual prompting for direct inference. However, the impact of these choices remains unclear. In this paper, we systematically evaluate the impact of different prompt translation strategies for classification tasks with RAG-enhanced LLMs in multilingual systems. Experimental results show that an optimized prompting strategy can significantly improve knowledge sharing across languages, therefore improve the performance on the downstream classification task. The findings advocate for a broader utilization of multilingual resource sharing and cross-lingual prompt optimization for non-English languages, especially the low-resource ones.
Introduction
Building global AI systems requires navigating the complexities of multilingual Retrieval-Augmented Generation (RAG). A common challenge arises when a Knowledge Base (KB) is primarily in English, but the end-user interacts in a different language, such as French or Hindi. Developers typically choose between direct inference (mixing languages) or pre-translating the entire prompt into a single language. This study investigates which specific components of a prompt—instructions, persona, or candidate labels—benefit most from translation and how these choices affect model performance.
Prompting Strategies and Methodology
The researchers decomposed the LLM prompt into four distinct components to evaluate the sensitivity of each to translation:
- Identity (I): The high-level persona assigned to the model.
- Rules (R): The specific constraints and instructions for the task.
- Candidates (C): The list of potential labels or answers retrieved from the KB.
- Utterance (U): The user's input, which is always maintained in the source language.
Six strategies were tested, ranging from a baseline (English instructions and candidates) to a fully localized prompt. The study utilized a dataset of 6,000 utterances in French (high-resource) and Hindi (low-resource) for an intent classification task involving 80 distinct classes.
Results and Analysis
The experimental results reveal a clear divide between high-resource and low-resource language performance. For Hindi, forcing the model to output in the source language (Strategy 'A' or 'C') often led to performance degradation, with some models seeing drops of over 30%. This suggests that many LLMs still rely on English as an "internal logic" language for complex reasoning.
However, a "hybrid" approach often yielded the best results. Translating the Identity and Rules (the task "framing") into the user's native language while keeping the Candidates in English (the I&R strategy) improved accuracy for several models. For instance, Qwen2.5-7B-Instruct showed a +6.7% improvement in Hindi using this method.
Model heritage also played a significant role. BLOOMZ, a model with a strong multilingual training background, demonstrated robust performance when prompts were localized, showing a +13.4% improvement in French under the "All" strategy. In contrast, models like Llama-3.1 and Mistral-Nemo generally performed better when the prompt remained closer to the English baseline.
Conclusion
The findings suggest that there is no one-size-fits-all approach to multilingual RAG prompting. While high-resource languages like French are more flexible, low-resource languages like Hindi benefit significantly from a hybrid strategy: localize the task framing to provide context, but retain technical labels and output requirements in English to leverage the model's internal strengths. This research highlights the importance of cross-lingual prompt optimization in making AI more accessible and effective globally.
Summary updated