A balloon effect: ban one channel and a hidden trait finds another

Keuntae Kim, Yong Suk ChoiarXiv 2026 · arXiv

TL;DR

Banning specific tokens (like an animal's name) to prevent a language model from passing on hidden traits is like squeezing a balloon: the trait simply "bulges" out through another channel, such as subtle statistical shifts in number lists or misspellings. This paper identifies the internal "trait vectors" and neural circuits responsible for this "balloon effect" and demonstrates that patching these specific circuits is far more effective at stopping trait transfer than surface-level token filtering.

What Problem Does This Paper Address?

As we increasingly train AI models on the outputs of other models (distillation), we face a stealthy problem: subliminal learning. Previous research showed that if a "teacher" model is prompted to love ravens, a "student" model trained only on the teacher's lists of numbers will somehow inherit that preference, even if the word "raven" never appears.

The intuitive defense is to find and forbid the tokens that carry the trait. But does this actually remove the trait from the data, or does it just force the model to find a new, perhaps more subtle, way to express it? This paper investigates the robustness of these hidden traits and whether surface-level bans are a futile game of "whack-a-mole."

Methods

The authors developed two "stress tests" to push models to their limits:

  1. Preference Chain: The model is asked to continue a list (e.g., numbers). The authors identify the most over-represented character (say, the digit '8' in the second position), ban it, and repeat. They track how long the "chain" of new over-represented features continues.
  2. Name Chain: They ban the entire vocabulary of a format (e.g., all digits for a number list). This usually forces the model to leak the trait's name directly. They then ban the name, then its fragments, observing where the "excess" trait signal goes next.

To understand why this happens, they used mechanistic interpretability techniques like activation patching to locate the specific "relay" and "identity" heads in the transformer architecture that write these traits into the output.

Fig. 1 — The balloon effect. a, A teacher told it loves snails writes only numbers, with no snail anywhere, yet a student trained on such numbers acquires the preference (shown in earlier work). The obvious defence is to forbid whatever carries the trait. b, A hidden trait acts like air in a balloon. Shown for a teacher told it loves snails and asked only for numbers: with no filter the trait tilts the numbers; banning every digit makes the name itself pour out; banning the name makes it misspell its way out; banning its first three letters leaves two letters carrying it. Quotes are actual outputs; gold numbers are shares of list items. The arrow u, the trait vector, keeps its direction and identity throughout. c, Projected through the output embedding, the trait vector raises some allowed tokens more than others (bars are schematic). Measured, this projection names the trait a leakage pattern belongs to; picking the single next channel is a harder target (top-1 18–31%).

Datasets

The study used two primary models: Qwen3-4B and Llama-3.2-3B. They tested 16 traits (8 rare animals and 8 emotions) across 6 non-text formats:

  • Three-digit numbers
  • Punctuation clusters
  • Mathematical operator clusters
  • Hexadecimal strings
  • DNA-like strings
  • One-line code assignments

Key Insights and Ideas

  • The Balloon Effect: Traits are not tied to specific tokens but are held as internal trait vectors. Forbidding one outlet just causes the vector to project onto the next most favorable allowed tokens.
  • Meaning Leaks into Statistics: In prompted models, traits are stored as concepts. For example, a model "loving" a large animal (like an elephant) will naturally produce larger numbers in a list compared to one "loving" a small animal (like an ant), even without mentioning the animal.
  • Prompting vs. Fine-tuning: There is a fundamental difference in how these traits are stored. Prompted traits are conceptual and robust (the balloon effect is strong). Fine-tuned traits (using standard recipes) are more like "slot-to-name" memories; banning the specific name fragment is much more effective at silencing them.
  • The Relay Circuit: The authors identified a tiny circuit—just 1.4% of heads in Qwen—that acts as a "relay," reading the trait from internal storage and writing it into the output separators (like commas and spaces).

Key Results

  • Persistence of Identity: Even under a "full-vocabulary ban" that makes the requested task impossible, the internal trait vector remains stable and can be used to identify which trait is being suppressed with near-perfect accuracy.
  • Whack-a-Mole in Action: Banning an animal's name led to the model producing fragments (e.g., "snai" for snail), then two-letter prefixes ("sn"), and eventually related prose in other languages (see Figure 4).
  • Circuit Patching Works: While a 14-feature token ban only reduced trait transfer to a student model by 26%, patching the relay circuit halved the transfer and cut overt name leaks by up to 7x.
  • Format Specificity: Different formats "read" different parts of the trait vector. DNA strings might reflect an animal's speed, while punctuation clusters might reflect its "arousal" or "size."
Fig. 1 — The balloon effect. a, A teacher told it loves snails writes only numbers, with no snail anywhere, yet a student trained on such numbers acquires the preference (shown in earlier work). The obvious defence is to forbid whatever carries the trait. b, A hidden trait acts like air in a balloon. Shown for a teacher told it loves snails and asked only for numbers: with no filter the trait tilts the numbers; banning every digit makes the name itself pour out; banning the name makes it misspell its way out; banning its first three letters leaves two letters carrying it. Quotes are actual outputs; gold numbers are shares of list items. The arrow u, the trait vector, keeps its direction and identity throughout. c, Projected through the output embedding, the trait vector raises some allowed tokens more than others (bars are schematic). Measured, this projection names the trait a leakage pattern belongs to; picking the single next channel is a harder target (top-1 18–31%).

Limitations and Future Work

  • Model Scale: The study focused on 3-4 billion parameter models. It remains to be seen if the "balloon" becomes even harder to squeeze in frontier models with hundreds of billions of parameters.
  • Trait Complexity: The traits used (animals, emotions) are relatively simple. Future work could explore more complex or abstract traits like political bias or specific reasoning styles.
  • Defense Completeness: While circuit patching is more effective than token banning, it only reduced student transfer by half. Finding a way to completely "de-bias" the teacher's output remains an open challenge.

Summary updated

Abstract

Language models trained on the outputs of other models can acquire traits that those outputs never mention, such as a preference for an animal carried by lists of numbers. Such subliminal learning can pass on unwanted traits. The intuitive defence is to find and forbid the tokens through which a trait shows, but how a trait's trace behaves under such bans, and what of it a student learns, has not been resolved. We closed a trait's outlets one by one in two stress tests, a preference chain and a name chain, across two models, six output formats and 16 traits. Bans did not remove the trait. Much as squeezing a balloon in one place makes it bulge in another, the trait moved to wherever the format still allowed, an effect we call the balloon effect. Decomposing this transfer, which bans on surface features cannot stop, down to single heads and neurons, we show that it can be cut at the circuit that writes the trait into the output. Patching that circuit while the teacher writes numbers stopped the chains, sharply reduced the overt name leak and halved the trait passed to a student, whereas the same number of random heads produced no detectable reduction. These results give a mechanistic account of how unwanted hidden traits are transferred and a route to preventing it.
BibTeX
@misc{kim2026balloon,
  author = {Keuntae Kim and Yong Suk Choi},
  title = {{A balloon effect: ban one channel and a hidden trait finds another}},
  year = {2026},
  eprint = {2611.02509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02509},
}
Figure