NatLan: Native Language Prompting Facilitates Knowledge Elicitation Through Language Trigger Provision and Domain Trigger Retention

Baixuan Li, Yunlong Fan, Zhiqiang Gao, Gao, Zhiqiang

Abstract

Multilingual large language models (MLLMs) often struggle to match their dominant-language performance when answering questions in non-dominant languages. While "translate-then-answer" methods offer a partial solution, the underlying mechanisms of their success—and their frequent failures in technical domains—have remained poorly understood. This paper introduces Native Language Prompting (NatLan), a framework that uses human cognitive analogies—Language Triggers (LT) and Domain Triggers (DT)—to interpret and improve multilingual knowledge elicitation. By employing a Multi-MLLM collaboration strategy with a domain-specific translator, NatLan achieves up to a 31.28% improvement in accuracy across five major languages, ensuring that critical technical terminology is retained during the translation process.

Introduction: The Multilingual Performance Gap

Despite being trained on vast multilingual corpora, state-of-the-art models like GPT-4o or Llama-3 exhibit a "native language" bias, performing significantly better in English (their dominant training language) than in others. A common workaround is to translate non-English queries into English before processing. However, this approach often stumbles on specialized or technical content. Standard translation tools frequently prioritize fluency over technical precision, replacing specific jargon with generic synonyms. This loss of precision prevents the model from accessing the specialized knowledge it possesses in its dominant language.

Conceptual Framework: Language and Domain Triggers

To address this, we propose two key concepts:

  1. Language Trigger (LT): The provision of a query in the model's dominant language, which reduces the "cognitive load" required for basic comprehension.
  2. Domain Trigger (DT): The retention of specific, fine-grained technical terms that act as keys to unlock specialized knowledge domains within the model's parameters.

As illustrated in Figure 1, standard Neural Machine Translation (NMT) might provide a sufficient LT but fail to retain the DT. For example, a chemistry question regarding "complete combustion" might be translated generically as "burned," leading the model to miss the technical nuances required for a correct answer.

Comparison of translation methods showing Language and Domain Trigger retention.

Methodology: The NatLan Framework

NatLan implements a Multi-MLLM collaboration strategy to ensure both LT provision and DT retention. The architecture consists of two primary components:

  • Translator LLM: A role-enhanced, domain-specific model (e.g., GPT-4o-mini or Qwen) that uses specialized instructions and few-shot examples to translate queries while strictly preserving technical DTs.
  • Speaker LLM: The target model (e.g., Phi-3, Gemma, Mistral) that receives the high-quality English translation and generates the final response.
The NatLan workflow featuring Translator and Speaker LLMs.

Experimental Evaluation

The researchers evaluated NatLan across five languages—Arabic, Chinese, French, German, and Japanese—using two comprehensive benchmarks:

  • Multilingual MMLU (MMMLU): 14,079 multiple-choice questions spanning 57 disciplines.
  • C-Eval: A Chinese-specific suite with 13,948 questions across 52 disciplines.

Key Results and Insights

NatLan consistently outperformed baselines such as "Self-Translation" and "Google-MT." Key findings include:

  • Accuracy Improvements: NatLan delivered accuracy gains of up to 31.28% compared to direct answering in the original language.
  • Superior DT Retention: In up to 87% of cases, NatLan provided better or comparable retention of domain-specific triggers compared to state-of-the-art NMT.
  • Knowledge Activation Alignment: Analysis of the Speaker LLM's internal hidden states revealed that NatLan's translations caused internal activations to more closely mirror those seen when the model answers human-written "gold standard" English questions. This suggests that NatLan effectively "rectifies" the model's internal knowledge retrieval process.

Limitations and Future Directions

While highly effective, NatLan faces challenges in disciplines where translation may introduce errors, such as Mathematics (formula corruption) or culturally specific subjects where the relevant knowledge may not exist in the English training set. Future work will explore adaptive language switching to dynamically determine the optimal language for eliciting specific knowledge.

Conclusion

NatLan demonstrates that the key to unlocking the full potential of multilingual LLMs lies not just in translation, but in the strategic retention of domain-specific triggers. By aligning machine prompting with human cognitive principles, NatLan provides a robust framework for high-precision multilingual knowledge elicitation.

Summary updated

Abstract

Multilingual large language models (MLLMs) do not perform as well when answering questions in non-dominant languages as they do in their dominant languages. Although existing translate-then-answer methods alleviate this issue, the mechanisms behind their effectiveness remain unclear. In this study, we analogize the dominant language of MLLMs to the native language of humans and use two human cognitive features: the Language Trigger (LT) and the Domain Trigger (DT), to interpret the mechanisms behind translate-then-answer methods. This reveals that while sufficient LTs are provided by these methods, there remains a deficiency in DT retention. To mitigate this issue, we propose Native Language Prompting (NatLan), employing a Multi-MLLM collaboration strategy and introducing an additional role-enhanced domain-specific MLLM with stronger multilingual understanding capabilities as the translator. Across five language QA benchmarks, NatLan achieves up to a 31.28% improvement in accuracy and, compared to existing state-of-the-art methods, provides comparable or greater retention of DTs in up to 87% of cases. Our code is available at https://github.com/AnonyNLP/NatLan.
BibTeX
@misc{li2024natlan,
  author = {Baixuan Li and Yunlong Fan and Zhiqiang Gao and Gao, Zhiqiang},
  title = {{NatLan: Native Language Prompting Facilitates Knowledge Elicitation Through Language Trigger Provision and Domain Trigger Retention}},
  year = {2024},
  eprint = {2408.03544},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.03544},
}
Figure