Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation

Yongxi Zhou, Junwei Yao, Wenbo Ye, Jiahui GearXiv 2026 · arXiv

This paper explores a nuanced dimension of AI safety: it’s not just what language a harmful request is written in, but which part of the request carries that language. By breaking down prompts into functional components, the authors reveal that current safety evaluations might be missing a critical "component assignment" effect.

TL;DR

The researchers introduce a "paired reversal" method to evaluate multilingual safety by independently translating the request (e.g., "Write a script") and the operation (e.g., "to exploit vulnerabilities") of a harmful prompt. They found that refusal rates can shift significantly depending on which component is translated, suggesting that safety guardrails are sensitive to where non-English text appears within a prompt.

What Problem Does This Paper Address?

Standard multilingual safety benchmarks typically translate an entire prompt from English into a target language. While useful, this "whole-prompt" approach is a blunt instrument. It doesn't tell us if a model's failure to refuse a harmful request is due to a lack of understanding of the harmful intent (the operation) or a failure to follow the instruction (the request) when presented in a different language.

Furthermore, real-world "red teaming" has shown that "code-switching" (mixing languages) can bypass safety filters. This paper seeks to formalize how these mixed-language prompts affect model behavior by isolating the linguistic assignment of specific prompt components.

Methods

The authors developed a factorial design centered on two "slots":

  1. Request Slot: The delivery verb group (e.g., "Provide step-by-step instructions").
  2. Operation Slot: The harmful content (e.g., "on how to build a dangerous device").

Everything else—the "scaffolding" of the sentence—is kept in English (the "matrix language"). They assigned three languages—English (E), Chinese (Z), and Spanish (S)—independently to these two slots, creating nine configurations (e.g., ZE means a Chinese request with an English operation).

To ensure a fair comparison, they used a "dose-balanced" approach. For example, they compared EZ (English request, Chinese operation) against ZE (Chinese request, English operation). Both have exactly one translated component, allowing the researchers to isolate the effect of which component was translated.

Safety was measured using two separate metrics scored by LLM judges (Claude-3-Haiku and Qwen-7B):

  • Refusal: Did the model decline the request?
  • Unsafe Compliance: Did the model actually provide harmful, operationally useful info?

Datasets

The study utilized a pool of 370 behaviors drawn from established benchmarks:

  • AdvBench and HarmBench: Sources for harmful behaviors.
  • XSTest: Used for 100 benign items to check for "over-refusal" (refusing safe requests).

After a rigorous "gating" process—where human and automated checks ensured the translated prompts preserved the original intent and remained natural—they settled on a core set of 190 harmful behaviors for the final analysis.

Table I: Breakdown of behaviors retained at each stage of the study.

Key Insights and Ideas

  • Component Assignment Matters: The "where" of a language matters as much as the "what." Translating the request often led to higher refusal rates than translating the operation, even when the total amount of non-English text was similar.
  • Refusal $\neq$ Safety: A lower refusal rate doesn't always mean more danger. The authors argue that refusal and unsafe compliance must be reported separately because a model might "fail" to refuse in a standard way while still not providing a useful harmful response.
  • Model-Specific "Blind Spots": The sensitivity to these swaps varied wildly between models. This suggests that safety training (fine-tuning) might be more linguistically lopsided than the general pre-training.
  • The "Length" Confound: In their dataset, operation components were significantly longer (avg. 14.3 characters in Chinese) than request components (avg. 6.1 characters). This suggests that "text quantity" might be a hidden factor in why certain components trigger guardrails more effectively.

Key Results

The researchers tested four models: Gemini 2.5 Flash, DeepSeek-chat, GPT-4o-mini, and Mistral Small 4.

  • Mistral Small 4 showed the most dramatic effect: a 13.7 percentage point difference in refusal when swapping Spanish between slots.
  • Gemini and DeepSeek showed smaller but consistent "refusal swaps" of about 5.3 points for Spanish.
  • GPT-4o-mini was the outlier, maintaining extremely high refusal rates (97%+) across almost all configurations, showing high robustness to these linguistic swaps.
Fig. 1. Primary-judge refusal rates (%) on the same 190 harmful behaviours: primary analysis above, exploratory extension below, with one colour scale. Rows assign the request slot’s language and columns the operation slot’s; E, S, Z denote English, Spanish, Chinese. Cell denominators are 190 except six Gemini cells with empty generations, where 𝑛= 187–189. Wilson 95% intervals are omitted; their maximum half-width across all 36 cells is 6.2 𝑝𝑝.

Limitations and Future Work

  • Small Sample Size: The final analyzed set of 190 behaviors is relatively small due to the strict filtering required to ensure high-quality, intent-preserving translations.
  • English-Centric Frame: The study kept the "matrix language" (the frame of the sentence) in English. Future work could explore how results change if the frame itself is translated.
  • High-Resource Bias: The study only looked at Chinese and Spanish. The authors note that extending this to low-resource languages would be much harder due to the difficulty of ensuring translation quality and intent preservation.
  • Disentangling Length: Future experiments should attempt to match the character length of request and operation slots to see if the "component effect" is purely structural or driven by the volume of non-English text.

Summary updated

Abstract

Whole-prompt translation evaluates a request in different languages but does not compare which component carries a language. We introduce a paired reversal: translate the complete request component or the complete operation component into the same target language, holding behaviour, English frame and translated-span count fixed. Component identity and non-English text quantity change together. Assigning English, Chinese and Spanish independently to the two components yields nine configurations on 190 of 300 sampled harmful behaviours retained after consensus annotation and automatic gating. Refusal scored by language-model judges is the primary outcome; unsafe compliance is reported separately. On Gemini 2.5 Flash and DeepSeek, three refusal swaps are about 5.3 percentage points and the fourth is -0.5; none survives the pre-specified six-test Holm correction. In an exploratory extension, Mistral Small 4 has a 13.7-point Spanish swap (95% interval [8.4, 18.9]), which survives the twelve-test Holm sensitivity correction under the primary judge, and 12.1 points under a second judge. GPT-5.4-mini estimates are within 2 points of zero. Component assignment supplies an additional evaluation dimension. Evaluations should record component languages and translated text quantity, comparing matched behaviours at equal translated-span count while interpreting refusal and unsafe compliance separately.
BibTeX
@misc{zhou2026component,
  author = {Yongxi Zhou and Junwei Yao and Wenbo Ye and Jiahui Ge},
  title = {{Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation}},
  year = {2026},
  eprint = {2610.38181},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38181},
}
Figure