Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation

Dongxu Wu, Sara Court, Micha ElsnerarXiv 2026 · arXiv

TL;DR

When external dictionaries provide translations that contradict an LLM’s internal training, which one wins? This paper introduces a controlled Mandarin-to-English translation task to show that LLMs—especially larger ones—overwhelmingly favor prompt-supplied "contextual knowledge" over their own internal "parametric knowledge," even when explicitly instructed to ignore the new rules.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used for translation in specialized domains or low-resource languages by providing them with external lexical resources (like dictionaries or glossaries) via prompting. However, these external definitions often conflict with the model’s prior knowledge. For example, a model might "know" that the Mandarin word 码农 (mǎnóng) means "programmer," but a prompt might define it as "apple."

This creates a lexical knowledge conflict. While we know these conflicts cause errors in real-world translation, we don't fully understand the factors that drive a model to choose one meaning over the other. This paper systematically investigates how word frequency, sentence context, prompt framing, and model size influence this tug-of-war between internal and external knowledge.

Methods

The authors constructed a controlled translation task using 150 two-character Mandarin nouns across five semantic categories (animals, fruits, vehicles, occupations, and musical instruments).

Key experimental variables included:

  • Controlled Redefinition: Assigning new English meanings to familiar Mandarin words (e.g., redefining "programmer" to mean "apple").
  • Word Frequency: Testing words across high, medium, and low frequency bands to see if "stronger" internal associations are harder to override.
  • Sentence Context: Providing sentences that either support the familiar meaning (PK-context) or the new meaning (CK-context).
  • Prompt Framing: Testing five different ways to present the rule, ranging from a bare dictionary entry to a "language game" rule, and even an explicit instruction to ignore the new rule.
Figure 1: Schematic illustration of lexical conflicts in Mandarin-to-English translation. The prompt assigns the new meaning ‘apple’ to the Mandarin word 码农 mˇanóng ‘programmer’. The model is then asked to translate sentences whose contexts favor either the fa- miliar meaning or the new meaning.

Datasets

  • Lexical Inventory: 150 Mandarin nouns selected from the BCC (Beijing Language and Culture University) corpus, categorized by frequency.
  • Evaluation Metrics: The primary metric is CK Reliance (the proportion of responses using the prompt-supplied meaning). They also used COMET and chrF++ to assess overall translation quality.

Key Insights and Ideas

  • The "Larger is More Compliant" Paradox: Within model families (e.g., Llama-3 8B vs. 70B), larger models are significantly more likely to adopt the new, prompt-supplied meaning. While this suggests better instruction-following, it also makes them more vulnerable to incorrect or malicious external data.
  • Context as a Catalyst: Models are far more likely to use a new meaning if the sentence context supports it. However, even when the sentence perfectly fits the original meaning, models still frequently "force" the new meaning into the output.
  • Frequency Matters: The more familiar a model is with a word (higher frequency), the less likely it is to be swayed by a conflicting prompt definition.
  • Framing is Power: How you present a rule changes its weight. A bare dictionary entry (e.g., "Lexicon: X means Y") is treated as more authoritative than a rule framed as a game.
  • Reasoning is Inconsistent: Enabling "thinking" or reasoning modes helped GPT-5.5 follow instructions to ignore a rule, but it actually drove Qwen models to adopt the new rule almost 100% of the time, regardless of instructions.

Key Results

  • Dominance of CK: Across Experiment 1, models used the supplied meaning in 75.07% of responses.
  • Instruction Failure: In the "Ignore" condition (F1), 7 out of 9 models still favored the new meaning more than 50% of the time.
  • Modal Weakening: Changing "means" to "may mean" reduced CK reliance by 17.73 percentage points, suggesting models are sensitive to the certainty of the provided information.
  • Positioning Over Repetition: Simply repeating a rule five times in the middle of a prompt didn't help. However, placing the rule both at the beginning and the end of the prompt (the "BE" layout) significantly boosted adoption.
Figure 2: Baseline familiarity rates by model. For each model, the left and right bars show the proportion of FAMILIAR responses in semantically consistent and in- consistent contexts, respectively.

Limitations and Future Work

The study relies on synthetic redefinitions (assigning "apple" to "programmer"), which are more extreme than the subtle domain-specific shifts found in real-world translation. The authors also note that their study was limited to nouns and a single language pair (Mandarin-English).

Future research could explore:

  • How models handle conflicts in low-resource languages where their internal knowledge is already weak.
  • Developing automated detection methods to flag when a retrieved dictionary entry might conflict with a model's high-confidence internal knowledge.
  • Investigating if these findings hold for other parts of speech, like verbs or adjectives.

Summary updated

Abstract

External lexical resources can help large language models (LLMs) translate texts in low-resource languages and specialized domains, but these resources may conflict with models' prior lexical knowledge. To isolate this conflict, we construct a controlled Mandarin-to-English translation task in which familiar source words are assigned new meanings within and across semantic categories. We treat these prompt-supplied meanings as contextual knowledge (CK) and systematically evaluate the effects of word frequency, sentence context, and prompt framing on the translated outputs. Across nine models, CK reliance is greater for less frequent source words and in sentence contexts that support the supplied meanings. Prompt framing can reduce CK reliance, but seven of nine models favor CK even when explicitly instructed to ignore the supplied meanings. Within model families, larger models are more likely than smaller models to utilize CK in their outputs. Three follow-up experiments further show that modal weakening lowers CK reliance, the effect of repetition depends on prompt layout, and the effect of reasoning varies across models. Overall, our controlled study reveals how CK reliance varies across conditions and models, motivating new methods to detect and evaluate lexical conflicts and guide models' lexical choices in LLM-based translation.
BibTeX
@misc{wu2026are,
  author = {Dongxu Wu and Sara Court and Micha Elsner},
  title = {{Are These Programmers Ripe Enough to Eat? Evaluating How LLMs Handle Lexical Knowledge Conflicts in Translation}},
  year = {2026},
  eprint = {2611.09785},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09785},
}
Figure