TL;DR
Even when an aligned language model refuses to answer controversial or biased questions, that suppressed information is already encoded in its representations—and can be extracted with simple linear probes from completely neutral prompts. Furthermore, these representations persist untouched from pre-trained base models and actively leak into implicit downstream decision-making, showing that alignment merely hides harmful beliefs rather than erasing them.
What Problem Does This Paper Address?
Modern language models (LMs) are instruction-tuned and aligned via reinforcement learning from human feedback (RLHF) to refuse requests deemed harmful, sensitive, or biased (such as asking for the "average IQ" of an occupation or country). Yet, adversarial "jailbreak" prompts routinely bypass these defenses.
While prior work has treated jailbreaking as a generative bypass and refusal as a shallow behavioral circuit, a crucial question remains: What actually happens to the refused information itself? Does instruction-tuning eliminate or reorient these controversial representations, or does it merely silence their overt generation? And if this knowledge remains latent, does it still influence the model's subtle, indirect downstream decisions?
Methods
The authors bridge generative jailbreaks and representational probing using three main steps:
- Probing Innocuous Hidden States: Instead of feeding a harmful query, they pass a completely neutral prompt into the aligned model:
"This document describes [entity]". They train linear ridge regression probes on the resulting residual-stream hidden states across layers to predict the numerical values the model generates when jailbroken. - Jailbreak Benchmarking: To generate label targets, they elicit numerical answers using two jailbreaking approaches: a 5-shot in-context learning (ICL) attack and a Machiavellian role-playing prompt ("AIM").
- Base-to-Instruct Probe Transfer: To test if these representations persist through alignment, they train linear probes on pre-trained base models (which answer directly without refusing) and apply those frozen probes to the hidden states of instruction-tuned models to predict jailbroken outputs.
- Behavioral Relevance via Pairwise Comparisons: To see if suppressed features guide decisions, they prompt models to make pairwise comparisons (e.g., "Which occupation has a higher IQ: [A] or [B]?"), fit a Bradley–Terry preference model to obtain latent entity rankings, and compute Spearman correlations against probe predictions.
Datasets
The authors evaluate four entity collections paired with sensitive or controversial numerical attributes (e.g., IQ, substance abuse rate, net worth, criminal history, attractiveness):
- Countries ($N = 222$): Country/territory names sourced from Encyclopædia Britannica.
- Occupations ($N = 939$): Standard occupational titles from the U.S. Department of Labor's O*NET 29.2 database.
- Political Figures ($N = 2,282$): Global political leaders (1989–2023) from the Political Leaders’ Affiliation Dataset.
- Synthetic Names ($N = 1,000$): Fictional person names generated via the OpenAI API (e.g., "the fictional person named John Smith").
Evaluations span three open-source model pairs: gemma-2-9b / -it, gemma-2-2b / -it, and Yi-6B / -Chat.
Key Insights and Ideas
- Latent beliefs emerge without prompting: A model does not need to be provoked to activate controversial associations. Passing an innocuous phrase like "This document describes [occupation]" spontaneously organizes hidden states along dimensions like IQ and substance abuse.
- Instruction-tuning does not relocate knowledge: Probes trained entirely on base models transfer directly to instruction-tuned models. This provides strong mechanistic support for the Superficial Alignment Hypothesis—alignment teaches the model when not to talk, but leaves the underlying pre-trained representation geometry virtually unaltered.
- Suppressed knowledge is actively utilized: The decoded representations are not inert "leftovers." When asked to compare two entities implicitly, the model's generative preferences strongly align with the probed traits, demonstrating that latent biases can invisibly steer downstream evaluations.
Key Results
- High Linear Decodability: On
gemma-2-9b-it, linear probes trained on innocuous activations predict jailbroken answers with Pearson correlations exceeding $r = 0.7$ for most occupation attributes, with some exceeding $0.9$. Decodability scales with model size (9B models outperformed 2B and 6B models). - Strong Base-to-Instruct Transfer: Probes trained on raw base models successfully predicted instruction-tuned jailbreak generations, especially for Countries and Occupations, demonstrating cross-model representational stability.
- Downstream Correlation: Probed features correlated strongly with Bradley–Terry ranking scores from pairwise comparisons—achieving Spearman $r = 0.824$ for Occupation Percent Women and $r = 0.719$ for Occupation IQ in
gemma-2-9b-it.
Limitations and Future Work
- Continuous/Numerical Scope: Probes were restricted to scalar, numerical attributes; the authors do not evaluate qualitative harmful knowledge (e.g., synthesizing dangerous substances or cyberattack scripts).
- Downstream Task Variety: Downstream behavioral impact was tested solely via pairwise comparisons rather than complex, open-ended generative workflows.
- Causality vs. Correlation: The methodology demonstrates representational access and behavioral correlation, but does not perform causal interventions (such as activation steering or patching).
- Decoding Labels: Ground-truth targets were derived strictly from greedy generations of specific jailbreaks rather than marginalized token distributions.
Summary updated