Abstract
Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.
Introduction
While Large Language Models (LLMs) are known to suffer from hallucinations and social biases, a more subtle and pervasive glitch exists in how they process context. We identify this as semantic leakage: a phenomenon where irrelevant semantic features from a prompt "leak" into the model's output in unintended, often stereotypical ways. For instance, if a prompt mentions a character likes the color yellow before asking for their profession, the model might disproportionately assign them the job of a "school bus driver."
This behavior suggests that models fail to maintain a boundary between background context and the target task, leading to outputs that feel forced or overly thematic compared to natural human language.
Methodology
To measure this phenomenon, we developed a framework centered on the Leak-Rate metric. Our approach compares two prompt types:
- Control Prompt: A neutral task (e.g., "His favorite food is").
- Test Prompt: The same task preceded by a "leaking concept" (e.g., "He likes koalas. His favorite food is").
We calculate semantic similarity between the model's output and the leaking concept using three automatic embedding methods: BERT-Score, SentenceBERT, and OpenAI Embeddings. A "leak" is recorded if the test generation (e.g., "eucalyptus leaves") is semantically closer to the concept ("koalas") than the control generation is. These automatic metrics were validated against human annotators to ensure reliability.
Our test suite consists of 109 prompts across categories like colors, animals, and idioms, and includes translations into Chinese and Hebrew to test cross-lingual persistence.
Key Findings
The Impact of Instruction Tuning
Perhaps the most striking discovery is that instruction tuning—the process used to create "chat" or "instruct" versions of models—actually exacerbates semantic leakage. As shown in our analysis, chat-tuned models consistently exhibit higher Leak-Rates than their base counterparts. This may be because these models are trained to be highly responsive to prompt details, leading them to over-index on irrelevant information in an attempt to be "helpful" or creative.
Ubiquity and Model Performance
Semantic leakage is not limited to smaller models; it is present across all 13 flagship models tested, including GPT-4o, which was found to be particularly prone to the behavior. Leak-Rates across the board significantly exceeded the 50% random-chance threshold, often reaching 70-80%.
Language and Temperature Sensitivity
The phenomenon is language-independent, appearing in Chinese and Hebrew even when the leaking concept is provided in a different language than the generation task. Furthermore, for Llama-based models, lower temperatures (more deterministic sampling) generally correlate with higher leakage rates, suggesting that greedy decoding paths are more susceptible to these semantic pulls.
Conclusion and Future Work
Semantic leakage represents a unique form of bias that can hinder model logic and produce nonsensical associations. While our study characterizes the problem across diverse models and languages, it also highlights the difficulty of measuring leakage in long-form text where concepts might appear naturally. Future research must focus on mitigation strategies, such as architectural changes to attention mechanisms or specialized fine-tuning, to help models better distinguish between relevant instructions and irrelevant background context.
Summary updated