Large Language Models (LLMs) are often marketed as "multilingual," yet they frequently exhibit a startling "consistency gap." A model might correctly identify a medical fact or a historical date when asked in English, but provide a contradictory answer or a hallucination when asked the same question in Hindi, Arabic, or Chinese. This phenomenon, known as cross-lingual factual inconsistency, reveals that a model’s internal knowledge is often siloed by language rather than stored as a universal, language-agnostic concept.
The Anatomy of the Consistency Gap
The core problem is not necessarily a lack of knowledge, but a failure of retrieval and alignment. As Ashim Gupta, Maitrey Mehta, and Zhichao Xu demonstrate in Found in Translation, models become significantly less informative and empathetic in non-Latin scripts. This isn't just a matter of translation quality; Nadezhda Chirkova and Vassilina Nikoulina found a distinct decoupling between linguistic fluency and factual accuracy. A model can be perfectly grammatical in a target language while simultaneously failing to retrieve the correct facts it clearly "knows" in English.
This disparity is particularly dangerous in high-stakes domains. In Better to Ask in English, Yiqiao Jin and colleagues show that healthcare queries in languages like Hindi often return contradictory or unverifiable advice compared to English counterparts. Similarly, Varun Gumma and Sunayana Sitaram identified a "performance tax" in multilingual retrieval-augmented generation (RAG) systems, where the ability to synthesize retrieved information into a correct response is not uniform across languages.
Why Knowledge Fails to Transfer
Research suggests three primary mechanisms behind these failures:
- Entity Misalignment: Models often fail to map different language-specific names for the same entity to a shared internal concept. Yihong Liu and Hinrich Schuetze argue in On the Entity-Level Alignment in Crosslingual Consistency that if a model cannot recognize that "London" and "Londres" refer to the same entity, it cannot share facts between those linguistic contexts. Masaki Sakata and Benjamin Heinzerling further show that models organize these entities into linear subspaces early in their layers, and poor organization here leads directly to factual recall failure.
- The Language Transition Bottleneck: Internal probing reveals that models often process facts in a language-neutral "concept space" in their middle layers but stumble during the final "language transition" phase. Mingyang Wang and co-authors in Lost in Multilinguality found that factual errors often occur when the model tries to map these internal concepts back to a specific target language.
- High Response Variance: The knowledge may be present in the model's parameters but is difficult to retrieve stably in non-English languages. Vihari Piratla and Purvam Jain suggest that the cross-lingual gap is often a retrieval issue; when multiple outputs are sampled and aggregated, the "hidden" correct answer often emerges.
Mending the Multilingual Pipeline
To bridge this gap, researchers are moving beyond simple fine-tuning toward structural and algorithmic interventions.
- Internal Steering: Lu Meng and Carsten Eickhoff developed training-free vector interventions that steer the model’s internal pathways toward English-centric retrieval circuits, boosting accuracy in low-resource languages by over 35%. Similarly, Chenghao Xiao and Hou Pong Chan discovered that a model's "knowledge boundary"—its awareness of what it doesn't know—is encoded in a linear geometric structure that can be projected from English to other languages to reduce hallucinations.
- Entity-Based Augmentation: Strategies like LEIA, proposed by Ikuya Yamada and Ryokan Ri, inject English entity names into target-language training data to "unlock" English-centric knowledge for other languages.
- Strategic Alignment: Rather than using massive, noisy datasets, Weihao Liu and Ning Wu show in Selected Languages are All You Need that fine-tuning on a small subset of "core" languages can transfer truthfulness more effectively. Amit Agarwal and Sujith Ravi further suggest "batch-wise alignment," ensuring training batches contain semantically identical content across languages to force the model to build shared representations.
Ultimately, achieving cross-lingual consistency requires treating multilinguality not as a translation task, but as a challenge of knowledge synchronization. As Bryan Li argues, building models that are truly knowledgeable in a multilingual world requires moving past the "English-first" paradigm toward systems where facts remain invariant, regardless of the surface language used to query them.