Safety-trained language models often refuse a dangerous request in English and then answer the same request in Yoruba or Bengali. The leading explanation is that the model never recognizes the low-resource prompt as harmful in the first place. Refusal Direction is Universal Across Safety-Aligned Languages found that the internal "refuse" signal is one direction shared across 14 languages. Yet when harmful and benign Yoruba prompts are projected onto their principal components, they overlap into a "mushy" cloud, while English prompts split into two clean clusters. Llama 3.1 complies with 82.9% of harmful Yoruba prompts on PolyRefuse even without any intervention. That analysis, like most safety probing, reads one summary vector per prompt, usually the last input token at a chosen layer. We propose to test whether that single read discards the signal. The harmful intent may be present but scattered across particular tokens, layers, or the first few generated tokens.
The methodological lens comes from biology. In situ hybridization is a family of lab techniques that detect a specific RNA molecule inside intact tissue by letting a labeled probe bind to it where it sits. It exists because grinding tissue into a bulk sample and averaging can hide a transcript that is abundant in one small compartment. We carry over two concrete ideas and no biological mechanism. The first is to measure where a signal lives before concluding it is absent. The second, taken from padlock probes that only produce signal when both ends bind, is that requiring two independent detectors to agree keeps false alarms low. The claim is testable either way. If harmful intent in low-resource languages can be separated at identifiable sites, a small detector that reads those sites and switches on the universal refusal direction should close much of the cross-lingual gap without the over-refusal that blanket refusal amplification causes. If no site separates it, the failure is truly upstream and only data or translation can fix it. That negative answer would itself redirect the field.
Core Claim
Current evidence that low-resource harmfulness is "not encoded" rests on one pooled readout per prompt. We hypothesize that for medium- and low-resource languages the harmful/benign distinction is linearly readable, from a probe trained only on English, at specific (layer, token-position) sites that the last-input-token readout misses. Candidate sites include content-word tokens mid-prompt and the first generated tokens. The second possibility is motivated by Yuan et al., who showed that refusal can be triggered partway through a generation and that this ability generalizes across languages.
If the hypothesis holds, a gated intervention should follow directly. It would read harmfulness at the best sites and inject the universal refusal direction only when two detectors at different sites agree. That intervention should approach the safety of trained methods while keeping over-refusal near the unmodified model's level, with no weight changes and no target-language training data.
Relation to Prior Work
Two strands of prior work motivate the proposal. The first is the upstream/downstream split in Refusal Direction is Universal Across Safety-Aligned Languages: the refusal trigger transfers, but recognition of harmfulness does not. We reuse its difference-in-means refusal direction as the effector. We differ in how we test the "mushy" claim: position by position and layer by layer, instead of with one PCA view.
The second strand is evidence that harmful concepts are language-agnostic somewhere inside the network. Li et al. found that toxic concepts in MLP layers are shared across languages. Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations showed that knowledge boundaries are linearly encoded across languages. Together, these make it plausible that a shared harmfulness signal exists but is read from the wrong place.
Three methods constrain the design and serve as competitors, because they fix the gap by changing weights. Align Once, Benefit Multilingually (MLC) trains with a loss that forces translated prompts to share one direction, and it needs parallel translations. Who Transfers Safety? finds that fewer than 0.3% of neurons carry cross-lingual safety and fine-tunes around them. Multilingual Safety Alignment Via Sparse Weight Editing computes a closed-form edit that maps low-resource harmful representations into the English safety subspace, restricted so that benign activations are untouched. Our approach changes no weights and asks a prior question: is editing needed at all, or does reading from better sites suffice?
Xu et al. found that concept vectors transfer from dominant to low-resource languages but not the reverse. That asymmetry predicts that English-trained probes should transfer to other languages, and it is why we train probes on English. The biology papers play a narrow role. Systematic benchmarking of imaging spatial transcriptomics platforms in FFPE tissues evaluates detection by per-target sensitivity at controlled false discovery rate, and checks it against an orthogonal measurement (single-cell sequencing). We adopt exactly that reporting discipline. The per-language detection map is the analogue of per-gene sensitivity, and translation-based judgments serve as the orthogonal check.
Research Program
The first stage builds a detection map. For each model (Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Gemma-2-9B-it) we run the 572 harmful and 572 benign PolyRefuse prompts in all 14 languages. We record residual-stream activations at every layer and every input token, plus the first 8 generated tokens under greedy decoding. At each (layer, position) site we train a logistic probe on English only. We then report, per language, the AUROC and the sensitivity at a 5% false positive rate.
Positions must be comparable across languages that tokenize very differently. We therefore index them relatively (fraction of prompt length) and by role (content tokens, chat-template tokens, generated tokens). The baseline readout, the last input token at the refusal-direction layer, is one cell of this map. The scientific result is the gap between that cell and the best cell for each language.
The second stage separates "encoded but not routed" from "not encoded." A high-scoring site could come from surface features such as dangerous loanwords or English named entities in the prompt. To check, we repeat the map on a control split in which such tokens are replaced by native-script paraphrases. We also test whether the best site's probe direction is causally upstream of refusal. The test adds the probe direction at that site and measures movement along the refusal direction at later layers.
The third stage builds the gated intervention. Two probes at sites chosen on a held-out English and German development set, at different layers or positions, must both exceed their thresholds. When they do, the universal refusal direction is added at its layer for the remainder of generation. For sites in generated tokens, this becomes a mid-generation refusal of the kind Yuan et al. trained, but triggered without training. The two-detector rule is the padlock idea. Its purpose is to cut false alarms on benign low-resource prompts, which the ablations isolate.
We report a primary setting with zero target-language labels. A secondary setting allows 32 labeled target-language examples for choosing thresholds only, to show how much of any residual gap is calibration.
Baselines
The naive reference is the unmodified instruction-tuned model, evaluated identically. It anchors both attack success and over-refusal. On PolyRefuse, Llama 3.1 already complies with 82.9% of harmful Yoruba prompts (Refusal Direction is Universal Across Safety-Aligned Languages). Beating it is necessary but not interesting.
The "bulk" probe baseline is the same gated intervention with a single probe at the last input token and the refusal-direction layer. It isolates the claim that spatial resolution matters. It gets the same English training data, the same threshold-selection data, and the same refusal injection. The proposal beats it only if the best-site probe improves low-resource detection AUROC by a clear margin and lowers attack success at matched over-refusal. We set that margin in advance at 0.05 AUROC for at least three of the medium- and low-resource languages.
Ungated refusal-direction amplification adds the refusal direction to every non-English prompt. It is the cheapest possible fix. It will be run at several strengths, and we compare at matched attack success. We beat it if our over-refusal on translated XSTest and on PolyRefuse benign prompts is substantially lower at the same safety level.
The strongest existing method, on the numbers available in the file, is SS-Neuron Expansion from Who Transfers Safety?. It reports average MultiJail attack success on Llama-3.1-8B-Instruct dropping from 29.44% to 3.41%, and about 6% in Bengali where standard models fail over 60% of the time. MLC (Align Once, Benefit Multilingually) is a close second. It reports an average safety rate of 95.94% on Qwen-2.5-7B-Instruct against 66.44% for plain DPO on its translated PKU-SafeRLHF evaluation. Sparse Weight Editing reports a 27.33% average attack-success reduction on Multi-StrongREJECT for Llama-3.2-1B.
We rerun SS-Neuron Expansion and Sparse Weight Editing on the same model checkpoints, using their released settings with a matched hyperparameter search budget. We do the same for MLC on Qwen-2.5-7B, and we score all methods with the same judge on MultiJail, Multi-StrongREJECT, and PolyRefuse. Our method uses no training, so "beating" these methods has a specific meaning. We count it as a win if attack success is within 2 points of SS-Neuron Expansion, with lower over-refusal and no utility loss on MGSM and M-MMLU. A statistically lower attack success rate would be a strong win.
The translate-then-guard pipeline translates the input to English and applies an English guard classifier. There is no reported result found on Lacuna for translate-then-guard on these benchmarks, so we run it ourselves with the same judge and report its latency.
Two ablations isolate the key ingredients. The single-best-site ablation drops the second detector, which tests whether two-detector agreement matters. The input-only ablation excludes generated-token sites, which tests whether recognition emerges only once the model starts answering.
Why not just…?
Why not just train one multilingual probe at the usual layer and gate on it? This is the most serious threat. If a single last-token probe trained on a few translated examples matches our best-site probe, the spatial map adds nothing practical. The bulk-probe baseline and the 32-shot secondary setting test this directly. The Yoruba PCA result predicts the single probe will fail on low-resource languages, but PCA is unsupervised and a supervised probe could succeed where it fails. We will report it plainly if that happens.
Why not just translate to English and use an English guard? It adds a translation model and latency on every request. Its quality is bounded by translation quality in exactly the languages where translation is weakest, and Multilingual Jailbreak Challenges in Large Language Models shows that multilingual formatting still enables intentional jailbreaks. Still, no number exists for it here, so we measure it. If it wins on low-resource languages, our contribution narrows to a latency-free alternative for medium-resource languages.
Why not just fine-tune, as MLC or SS-Neuron Expansion do? These methods already reach strong safety. Fine-tuning is not free, though: full fine-tuning dropped Gemma-2 MMLU from 79% to 12% (Who Transfers Safety?). MLC needs parallel translations. Sparse methods concentrate safety in few neurons, which Who Transfers Safety? warns makes it easy to remove. Our method leaves weights untouched and can sit on top of any of them. We test that stacking explicitly by adding the gated intervention to an SS-Neuron-edited model.
Why not just amplify the refusal direction for non-English inputs? If recognition fails upstream, amplification cannot distinguish harmful from benign prompts, so safety is bought with over-refusal. The ungated baseline measures this trade-off curve directly.
Decisive Checks
The first check is the detection map itself. If, for Yoruba, Swahili, Javanese, and Bengali, no site exceeds the last-token readout by 0.05 AUROC under English-only training, and the paraphrase control removes whatever gains remain, then we conclude the upstream failure is real. We then stop at the analysis paper, which still settles an open question.
If some sites do exceed it, the second check is causal. Steering along the best-site probe direction must move downstream activations along the refusal direction. If it does not, the signal is readable but unused, and gating remains a fix applied from outside the model's own processing.
The third check is behavioral. The gated intervention must be compared with SS-Neuron Expansion's 3.41% average MultiJail attack success on Llama-3.1-8B, at over-refusal no worse than the unmodified model on translated XSTest, with unchanged MGSM and M-MMLU.
Finally, following the spatial-transcriptomics benchmark's use of an orthogonal reference, every detection claim is also scored against harmfulness labels obtained by translating prompts to English and judging them there. This guards against probes that learn translation artifacts rather than intent.
References
- Refusal Direction is Universal Across Safety-Aligned Languages
- Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
- Preference Tuning For Toxicity Mitigation Generalizes Across Languages
- Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations
- Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment
- Who Transfers Safety? Identifying and Targeting Cross-Lingual Shared Safety Neurons
- Multilingual Safety Alignment Via Sparse Weight Editing
- Exploring Multilingual Concepts of Human Values in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?
- Systematic benchmarking of imaging spatial transcriptomics platforms in FFPE tissues
- Multilingual Jailbreak Challenges in Large Language Models