Large Language Models (LLMs) possess two distinct mechanisms for processing information: parametric memory, which stores vast statistical associations acquired during pre-training, and contextual adaptation, which allows the model to reason over immediate inputs via In-Context Learning (ICL).
When these two systems align, LLMs appear remarkably intelligent. However, when a task requires a model to act against its pre-training—such as mapping the word "good" to a negative sentiment label, or adopting a counter-intuitive rule—a fundamental tension emerges. This phenomenon, known as semantic prior interference, reveals that LLMs frequently rely on semantic shortcuts rather than true abstract reasoning, prioritizing ingrained vocabulary biases over explicit instructions.
The Illusion of In-Context Learning
To understand why semantic priors interfere, we must look at how LLMs interpret few-shot demonstrations. Ideally, an LLM should analyze the structural relationship between an input and an output token, infer the underlying rule, and apply it to a query.
In practice, models often bypass this relational reasoning entirely. As Sophia Ananiadou et al. point out in How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning, evaluating ICL is deeply confounded by the model's pre-existing knowledge of label names. Instead of learning a novel mapping from the prompt, the model often uses the semantic meaning of the label words as an anchor.
When researchers intentionally disrupt these anchors, the illusion of learning dissolves. In Symbol tuning improves in-context learning in language models, Jerry W. Wei et al. demonstrate that models struggle significantly when labels are reversed or contradicted by the prompt. Because the model's weights are optimized to associate specific words with specific concepts, it cannot easily suppress those associations in favor of temporary, in-context evidence. This reliance on semantic priors, as noted by Cheng Bing et al. in Improving Input-label Mapping with Demonstration Replay for In-context Learning, prevents the model from effectively utilizing demonstrations that employ arbitrary or unrelated labels to define a task.
The Paradox of Scale and Overconfidence
Intuitively, one might expect that scaling a model's parameters or applying instruction tuning would make it better at following strict, unconventional rules. Paradoxically, the opposite is often true.
Larger models develop denser, more rigid internal representations, which can lead to a phenomenon known as inverse scaling. Bowen Li et al. explore this in An Investigation of LLMs' Inefficacy in Understanding Converse Relations, finding that when models are instructed to invert standard definitions (e.g., redefining "X contains Y" to mean "Y contains X"), larger models actually perform worse than smaller ones. Their stronger internal priors make it harder to override common-sense knowledge.
Similarly, Jason Wei et al. in Larger language models do in-context learning differently observe that while massive models eventually develop the emergent capacity to override priors given enough evidence, instruction tuning can reinforce internal semantic priors, making the model less flexible when confronted with unconventional input-label mappings.
[Standard Prompting]
"Great movie!" -> Positive (Prior Aligned) ✅ High Accuracy
[Verbalizer Manipulation / Flipped Labels]
"Great movie!" -> Negative (Prior Conflicting) ❌ Model defaults to Prior
This conflict also distorts the model's self-assessment. In Instruction-following Evaluation through Verbalizer Manipulation, Shiyang Li et al. use "verbalizer manipulation"—forcing models to use unnatural or neutral labels—to show that instruction-following drops sharply when labels conflict with semantic meaning. When this happens, the model's confidence calibration breaks down entirely. Youliang Yuan et al. demonstrate in Does ChatGPT Know That It Does Not Know? Evaluating the Black-Box Calibration of ChatGPT that prior-instruction conflicts cause a negative correlation between confidence and accuracy. The model outputs incorrect answers based on its pre-trained priors, yet expresses high confidence because the internal semantic associations remain strong.
Quantifying the "Semantic Baggage"
To map out where these decision boundaries fail, researchers use adversarial perturbations to isolate semantic biases from structural understanding.
- Label-Semantic Perturbation: Hila Gonen et al. in Prompting Language Models for Linguistic Structure prompt models using existing linguistic labels assigned to incorrect categories. The resulting performance drop reveals that the model's pre-trained structural knowledge is highly rigid and resistant to contextual redefinition.
- The Novelty vs. Adversarial Gap: In WinoDict: Probing language models for in-context word acquisition, Julian Martin Eisenschlos et al. show that assigning a novel definition to an existing, familiar word is significantly harder for an LLM than learning an entirely synthetic, nonsense word. The pre-existing weights for common vocabulary act as a stubborn anchor that prompting alone struggles to suppress.
- Cognitive Overload: When a task introduces multiple counter-intuitive meanings at once, the interference compounds. Arkil Patel et al. demonstrate in MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations that this "semantic baggage" severely limits an LLM's ability to fluidly adapt to adversarial definitions during inference.
This boundary distortion is visualized by Siyan Zhao et al. in Probing the Decision Boundaries of In-context Learning in Large Language Models, showing that the inherent semantic meanings of tokens create jagged, irregular decision boundaries that warp the logical patterns a model is expected to learn.
Pervasiveness Across Modalities and Domains
Semantic prior interference is not confined to text classification; it degrades performance across specialized domains, code generation, and even computer vision.
Code and Formal Logic
When learning new symbolic systems or code libraries on the fly, LLMs frequently default to standard programming conventions found in their training data. Arkil Patel et al. find in Evaluating In-Context Learning of Libraries for Code Generation that models struggle to adhere to new API constraints when those constraints conflict with entrenched coding habits. Similarly, Houfeng Wang et al. show in Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion that if a formal function name is counter-intuitive or conflicts with natural language priors, the model's ability to perform formal conversion drops significantly.
Knowledge Graphs and Optimization
In structural reasoning tasks, pre-trained facts often override logical rules. Yusuke Sakai et al. demonstrate in Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? that when models are evaluated on anonymized or shuffled knowledge graphs, their performance degrades, proving that they rely on memorized textual clues rather than structural inference. When applied to mathematical optimization, Pei-Fu Guo et al. find in Towards Optimizing with Large Language Models that models fail to execute specific meta-heuristic algorithms if the instructions deviate from standard mathematical principles embedded in their parametric memory.
Vision-Language Generation
The same interference affects text-to-image diffusion models. In Divide & Bind Your Attention for Improved Generative Semantic Nursing, Margret Keuper et al. note that strong internal priors can override requested attribute-object pairings if the combination is rare or counter-intuitive (e.g., a blue banana). Idan Schwartz et al. corroborate this in Discriminative Class Tokens for Text-to-Image Diffusion Models, showing that generative models possess rigid environmental associations that resist external steering when resolving lexical ambiguities.
Breaking Free from the Semantic Trap
To build models capable of true, flexible reasoning, researchers are developing techniques designed to decouple structural logic from token semantics.
Standard Fine-Tuning:
"I loved it" -> Positive (Reinforces label semantics)
Symbol Tuning:
"I loved it" -> Foo (Forces abstract mapping)
One prominent solution is Symbol Tuning, introduced by Jerry W. Wei et al. By stripping natural language labels from fine-tuning data and replacing them with arbitrary symbols (e.g., "foo/bar"), the model is forced to learn the task structure from the context rather than relying on label semantics.
For smaller architectures, such as visual language models, Mohammad Mahdi Derakhshani et al. introduce Self-Context Adaptation (SeCAt) in Small Visual Language Models can also be Open-Ended Few-Shot Learners. By training models on interleaved sequences with nonsense labels, they actively teach the network to suppress its semantic priors and focus on immediate contextual relationships.
Other approaches focus on dynamic, inference-time guidance. Chaoxu Pang et al. develop Guideline Learning in Guideline Learning for In-Context Information Extraction, which allows models to iteratively generate and retrieve explicit natural language rulebooks when navigating complex classification hierarchies. Similarly, Michal Stefánik et al. demonstrate in Concept-aware Data Construction Improves In-context Learning of Language Models that constructing training data based on shared latent reasoning concepts, rather than surface-level text patterns, helps mitigate the shortcut learning that makes semantic priors so dominant.
Ultimately, addressing semantic prior interference is essential for moving beyond models that merely echo their training data, paving the way for systems that can genuinely learn, adapt, and follow instructions in novel environments.
Key Foundational Work
WinoDict: Probing language models for in-context word acquisition (2023) — contributed 4 concept elements to this direction.
Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024) — contributed 3 concept elements to this direction.
MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations (2023) — contributed 3 concept elements to this direction.