When Pre-Trained Knowledge Conflicts with In-Context Instructions

46 concepts27 papers1 proposal

Large Language Models (LLMs) possess two distinct mechanisms for processing information: parametric memory, which stores vast statistical associations acquired during pre-training, and contextual adaptation, which allows the model to reason over immediate inputs via In-Context Learning (ICL).

When these two systems align, LLMs appear remarkably intelligent. However, when a task requires a model to act against its pre-training—such as mapping the word "good" to a negative sentiment label, or adopting a counter-intuitive rule—a fundamental tension emerges. This phenomenon, known as semantic prior interference, reveals that LLMs frequently rely on semantic shortcuts rather than true abstract reasoning, prioritizing ingrained vocabulary biases over explicit instructions.

The Illusion of In-Context Learning

To understand why semantic priors interfere, we must look at how LLMs interpret few-shot demonstrations. Ideally, an LLM should analyze the structural relationship between an input and an output token, infer the underlying rule, and apply it to a query.

In practice, models often bypass this relational reasoning entirely. As Sophia Ananiadou et al. point out in How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning, evaluating ICL is deeply confounded by the model's pre-existing knowledge of label names. Instead of learning a novel mapping from the prompt, the model often uses the semantic meaning of the label words as an anchor.

When researchers intentionally disrupt these anchors, the illusion of learning dissolves. In Symbol tuning improves in-context learning in language models, Jerry W. Wei et al. demonstrate that models struggle significantly when labels are reversed or contradicted by the prompt. Because the model's weights are optimized to associate specific words with specific concepts, it cannot easily suppress those associations in favor of temporary, in-context evidence. This reliance on semantic priors, as noted by Cheng Bing et al. in Improving Input-label Mapping with Demonstration Replay for In-context Learning, prevents the model from effectively utilizing demonstrations that employ arbitrary or unrelated labels to define a task.

The Paradox of Scale and Overconfidence

Intuitively, one might expect that scaling a model's parameters or applying instruction tuning would make it better at following strict, unconventional rules. Paradoxically, the opposite is often true.

Larger models develop denser, more rigid internal representations, which can lead to a phenomenon known as inverse scaling. Bowen Li et al. explore this in An Investigation of LLMs' Inefficacy in Understanding Converse Relations, finding that when models are instructed to invert standard definitions (e.g., redefining "X contains Y" to mean "Y contains X"), larger models actually perform worse than smaller ones. Their stronger internal priors make it harder to override common-sense knowledge.

Similarly, Jason Wei et al. in Larger language models do in-context learning differently observe that while massive models eventually develop the emergent capacity to override priors given enough evidence, instruction tuning can reinforce internal semantic priors, making the model less flexible when confronted with unconventional input-label mappings.

[Standard Prompting]
"Great movie!" -> Positive (Prior Aligned) ✅ High Accuracy

[Verbalizer Manipulation / Flipped Labels]
"Great movie!" -> Negative (Prior Conflicting) ❌ Model defaults to Prior

This conflict also distorts the model's self-assessment. In Instruction-following Evaluation through Verbalizer Manipulation, Shiyang Li et al. use "verbalizer manipulation"—forcing models to use unnatural or neutral labels—to show that instruction-following drops sharply when labels conflict with semantic meaning. When this happens, the model's confidence calibration breaks down entirely. Youliang Yuan et al. demonstrate in Does ChatGPT Know That It Does Not Know? Evaluating the Black-Box Calibration of ChatGPT that prior-instruction conflicts cause a negative correlation between confidence and accuracy. The model outputs incorrect answers based on its pre-trained priors, yet expresses high confidence because the internal semantic associations remain strong.

Quantifying the "Semantic Baggage"

To map out where these decision boundaries fail, researchers use adversarial perturbations to isolate semantic biases from structural understanding.

This boundary distortion is visualized by Siyan Zhao et al. in Probing the Decision Boundaries of In-context Learning in Large Language Models, showing that the inherent semantic meanings of tokens create jagged, irregular decision boundaries that warp the logical patterns a model is expected to learn.

Pervasiveness Across Modalities and Domains

Semantic prior interference is not confined to text classification; it degrades performance across specialized domains, code generation, and even computer vision.

Code and Formal Logic

When learning new symbolic systems or code libraries on the fly, LLMs frequently default to standard programming conventions found in their training data. Arkil Patel et al. find in Evaluating In-Context Learning of Libraries for Code Generation that models struggle to adhere to new API constraints when those constraints conflict with entrenched coding habits. Similarly, Houfeng Wang et al. show in Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion that if a formal function name is counter-intuitive or conflicts with natural language priors, the model's ability to perform formal conversion drops significantly.

Knowledge Graphs and Optimization

In structural reasoning tasks, pre-trained facts often override logical rules. Yusuke Sakai et al. demonstrate in Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? that when models are evaluated on anonymized or shuffled knowledge graphs, their performance degrades, proving that they rely on memorized textual clues rather than structural inference. When applied to mathematical optimization, Pei-Fu Guo et al. find in Towards Optimizing with Large Language Models that models fail to execute specific meta-heuristic algorithms if the instructions deviate from standard mathematical principles embedded in their parametric memory.

Vision-Language Generation

The same interference affects text-to-image diffusion models. In Divide & Bind Your Attention for Improved Generative Semantic Nursing, Margret Keuper et al. note that strong internal priors can override requested attribute-object pairings if the combination is rare or counter-intuitive (e.g., a blue banana). Idan Schwartz et al. corroborate this in Discriminative Class Tokens for Text-to-Image Diffusion Models, showing that generative models possess rigid environmental associations that resist external steering when resolving lexical ambiguities.

Breaking Free from the Semantic Trap

To build models capable of true, flexible reasoning, researchers are developing techniques designed to decouple structural logic from token semantics.

Standard Fine-Tuning: 
"I loved it" -> Positive  (Reinforces label semantics)

Symbol Tuning: 
"I loved it" -> Foo       (Forces abstract mapping)

One prominent solution is Symbol Tuning, introduced by Jerry W. Wei et al. By stripping natural language labels from fine-tuning data and replacing them with arbitrary symbols (e.g., "foo/bar"), the model is forced to learn the task structure from the context rather than relying on label semantics.

For smaller architectures, such as visual language models, Mohammad Mahdi Derakhshani et al. introduce Self-Context Adaptation (SeCAt) in Small Visual Language Models can also be Open-Ended Few-Shot Learners. By training models on interleaved sequences with nonsense labels, they actively teach the network to suppress its semantic priors and focus on immediate contextual relationships.

Other approaches focus on dynamic, inference-time guidance. Chaoxu Pang et al. develop Guideline Learning in Guideline Learning for In-Context Information Extraction, which allows models to iteratively generate and retrieve explicit natural language rulebooks when navigating complex classification hierarchies. Similarly, Michal Stefánik et al. demonstrate in Concept-aware Data Construction Improves In-context Learning of Language Models that constructing training data based on shared latent reasoning concepts, rather than surface-level text patterns, helps mitigate the shortcut learning that makes semantic priors so dominant.

Ultimately, addressing semantic prior interference is essential for moving beyond models that merely echo their training data, paving the way for systems that can genuinely learn, adapt, and follow instructions in novel environments.

Key Foundational Work

WinoDict: Probing language models for in-context word acquisition (2023) — contributed 4 concept elements to this direction.
Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024) — contributed 3 concept elements to this direction.
MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations (2023) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_prior_interference_56029d2a,
  title = {semantic prior interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-prior-interference/rdr_56029d2aef464d77ba84d9f653e5da91},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_prior_interference_56029d2a_v1,
  title = {semantic prior interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-prior-interference/rdr_56029d2aef464d77ba84d9f653e5da91/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake