TL;DR
Banning specific tokens (like an animal's name) to prevent a language model from passing on hidden traits is like squeezing a balloon: the trait simply "bulges" out through another channel, such as subtle statistical shifts in number lists or misspellings. This paper identifies the internal "trait vectors" and neural circuits responsible for this "balloon effect" and demonstrates that patching these specific circuits is far more effective at stopping trait transfer than surface-level token filtering.
What Problem Does This Paper Address?
As we increasingly train AI models on the outputs of other models (distillation), we face a stealthy problem: subliminal learning. Previous research showed that if a "teacher" model is prompted to love ravens, a "student" model trained only on the teacher's lists of numbers will somehow inherit that preference, even if the word "raven" never appears.
The intuitive defense is to find and forbid the tokens that carry the trait. But does this actually remove the trait from the data, or does it just force the model to find a new, perhaps more subtle, way to express it? This paper investigates the robustness of these hidden traits and whether surface-level bans are a futile game of "whack-a-mole."
Methods
The authors developed two "stress tests" to push models to their limits:
- Preference Chain: The model is asked to continue a list (e.g., numbers). The authors identify the most over-represented character (say, the digit '8' in the second position), ban it, and repeat. They track how long the "chain" of new over-represented features continues.
- Name Chain: They ban the entire vocabulary of a format (e.g., all digits for a number list). This usually forces the model to leak the trait's name directly. They then ban the name, then its fragments, observing where the "excess" trait signal goes next.
To understand why this happens, they used mechanistic interpretability techniques like activation patching to locate the specific "relay" and "identity" heads in the transformer architecture that write these traits into the output.
Datasets
The study used two primary models: Qwen3-4B and Llama-3.2-3B. They tested 16 traits (8 rare animals and 8 emotions) across 6 non-text formats:
- Three-digit numbers
- Punctuation clusters
- Mathematical operator clusters
- Hexadecimal strings
- DNA-like strings
- One-line code assignments
Key Insights and Ideas
- The Balloon Effect: Traits are not tied to specific tokens but are held as internal trait vectors. Forbidding one outlet just causes the vector to project onto the next most favorable allowed tokens.
- Meaning Leaks into Statistics: In prompted models, traits are stored as concepts. For example, a model "loving" a large animal (like an elephant) will naturally produce larger numbers in a list compared to one "loving" a small animal (like an ant), even without mentioning the animal.
- Prompting vs. Fine-tuning: There is a fundamental difference in how these traits are stored. Prompted traits are conceptual and robust (the balloon effect is strong). Fine-tuned traits (using standard recipes) are more like "slot-to-name" memories; banning the specific name fragment is much more effective at silencing them.
- The Relay Circuit: The authors identified a tiny circuit—just 1.4% of heads in Qwen—that acts as a "relay," reading the trait from internal storage and writing it into the output separators (like commas and spaces).
Key Results
- Persistence of Identity: Even under a "full-vocabulary ban" that makes the requested task impossible, the internal trait vector remains stable and can be used to identify which trait is being suppressed with near-perfect accuracy.
- Whack-a-Mole in Action: Banning an animal's name led to the model producing fragments (e.g., "snai" for snail), then two-letter prefixes ("sn"), and eventually related prose in other languages (see Figure 4).
- Circuit Patching Works: While a 14-feature token ban only reduced trait transfer to a student model by 26%, patching the relay circuit halved the transfer and cut overt name leaks by up to 7x.
- Format Specificity: Different formats "read" different parts of the trait vector. DNA strings might reflect an animal's speed, while punctuation clusters might reflect its "arousal" or "size."
Limitations and Future Work
- Model Scale: The study focused on 3-4 billion parameter models. It remains to be seen if the "balloon" becomes even harder to squeeze in frontier models with hundreds of billions of parameters.
- Trait Complexity: The traits used (animals, emotions) are relatively simple. Future work could explore more complex or abstract traits like political bias or specific reasoning styles.
- Defense Completeness: While circuit patching is more effective than token banning, it only reduced student transfer by half. Finding a way to completely "de-bias" the teacher's output remains an open challenge.
Summary updated