TL;DR
Large language models (LLMs) have a fascinating, unintended habit of letting completely irrelevant details from a prompt bleed into their generated outputs—a phenomenon the authors term semantic leakage. By evaluating 13 flagship models, the researchers demonstrate that this behavior is pervasive, systematically amplified by instruction tuning, and persists across multiple languages and open-ended generation tasks.
What Problem Does This Paper Address?
When you tell an LLM, "He likes yellow. He works as a...", a perfectly reasonable completion might be "teacher" or "engineer." Instead, flagship models like GPT-4o frequently output "school bus driver."
This paper identifies and characterizes semantic leakage: an undue influence where semantic features of unrelated words in a prompt heavily bias the model's generation. While standard biases (like gender or racial stereotypes) are well-documented, semantic leakage represents a much broader class of association bias. It can lead to forced, overwrought, or nonsensical outputs, and in adversarial or QA settings, it can cause "knowledge overshadowing"—where a strong association in a prompt overrides the actual logic required to answer a question correctly.
Methods
To catch and quantify semantic leakage, the authors designed a clean, comparative evaluation framework:
- Prompt Pairing: They paired a control prompt (lacking any spurious semantic signal, e.g., "His favorite food is") with a test prompt that injects an unrelated concept (e.g., "He likes koalas. His favorite food is").
- Semantic Leakage Rate (Leak-Rate): This metric calculates the percentage of times the model's generation is semantically closer to the injected concept than the control generation is. A baseline of 50% indicates random guessing (no leakage), while scores significantly higher than 50% flag active leakage.
- Similarity Measurement: The authors used three distinct embedding methods to automatically compute cosine similarities between the concept and the generations: BERT-Score (BS), SentenceBERT (SB), and OpenAI Embeddings (OAI).
- Human Validation: They verified the automated metrics by having native English speakers manually judge which generation was more semantically related to the target concept.
Datasets
The authors introduced a newly curated diagnostic test suite consisting of 109 manually created prompts. These prompts span diverse semantic categories, including colors, food, animals, songs, occupations, and idioms (e.g., testing if the literal meaning of "getting cold feet" leaks into the output).
To evaluate the breadth of this phenomenon, the dataset was extended in three ways:
- Multilingual & Crosslingual: Translated versions of the test suite in Chinese and Hebrew, as well as mixed-language prompts (e.g., Chinese context followed by an English completion hook).
- Open-Ended Stories: 23 prompts asking the model to write a short story about a child with a semantically loaded name (like Coral or Melody).
- Open-Ended Recipes: Prompts asking for a recipe to use with a specific colored pan (e.g., a blue pan).
Key Insights and Ideas
- Instruction Tuning Amplifies Leakage: One of the most striking findings is that chat- and instruction-tuned models (like Llama-3-8B-Instruct) consistently leak significantly more than their vanilla, pretrained base counterparts. The authors hypothesize that instruction tuning incentivizes models to provide more specific, colorful, and content-rich details, which inadvertently forces them to latch onto any available semantic hook in the prompt.
- Greedy Decoding Makes it Worse: For Llama models, lower sampling temperatures (especially greedy decoding at $t=0$) yield the highest semantic leakage rates. Higher temperatures introduce randomness that helps break these rigid semantic associations.
- Literal Idiom Bleeding: Models struggle to isolate the figurative meaning of idioms. Prompting a model with "She gave him the green light..." frequently causes the model to generate literal color descriptions (like "bright green") later in the text.
Key Results
The researchers evaluated 13 models across the GPT and Llama families, finding significant semantic leakage across the board:
- Pervasive High Leak-Rates: All tested models scored well above the 50% random mark. For instance, GPT-4o achieved an automatic Leak-Rate of 76.9% (BERT-Score) and 85.0% (OpenAI Embeddings).
- Human Agreement: Human annotators strongly validated these findings, scoring GPT-4o at a 71.8% Leak-Rate and Llama-3-70B-Instruct at 66.7%.
The clear gap between plain and instruction-tuned models across the Llama family can be seen in the automated benchmarks:
- Open-Ended Leakage: In story generation, giving a character the name Coral reliably caused GPT-4o to warp the entire narrative around coastal themes, beaches, crabs, and oceans, yielding an automatic Leak-Rate of up to 78.7%.
An example of how a single name can completely hijack the narrative direction of a story is illustrated below:
Limitations and Future Work
The primary limitation is the scale of the benchmark; the test suite relies on 109 manually curated prompts, as creating clean, non-confounding prompt pairs is difficult to automate. Additionally, automatic embedding metrics can occasionally introduce noise if the model simply repeats words from the prompt. Finally, due to Llama's performance limitations in Hebrew, multilingual analysis was heavily weighted toward Chinese and English.
For future work, the authors plan to formally investigate the mechanics of instruction tuning to see exactly how it incentivizes leakage. They also aim to explore the direct parallels between LLM semantic leakage and the psychological phenomenon of semantic priming in humans.
Summary updated