TL;DR
Strata-Sword introduces a hierarchical safety benchmark that evaluates Large Language Models (LLMs) and Large Reasoning Models (LRMs) against jailbreak attacks categorized by their underlying reasoning complexity. By testing 23 frontier and open-weight models across English and Chinese, the authors demonstrate that while current models reliably refuse basic harmful prompts, their safety guardrails frequently collapse when harmful intent is obscured behind multi-step logic, linguistic obfuscation, or cultural puzzles.
What Problem Does This Paper Address?
Most safety alignment techniques (such as RLHF or DPO) train models to detect and reject explicit, surface-level harmful requests (e.g., direct queries asking for illicit instructions). However, as models have become stronger at multi-step reasoning, this capability has become a double-edged sword: adversaries can disguise malicious intentions within complex instructions that require semantic deconstruction or code execution to uncover.
Existing safety benchmarks typically measure the breadth of harmful topics or push the brute-force strength of attack prompts, but they rarely measure at what level of cognitive reasoning a model's safety guardrails fail. Furthermore, safety evaluations are heavily skewed toward English, often ignoring how language-specific structural features—such as the logographic and semantic nature of Chinese characters—can be leveraged to bypass safety filters.
Methods
The authors quantify Reasoning Complexity along three foundational dimensions:
- Logical Depth: The number of multi-step, causal, or conditional inferences required to infer the core intent.
- Linguistic Ambiguity: The use of syntactic permutations, figurative language, or semantic obfuscation.
- Task Overhead: Extraneous cognitive load, such as code analysis or formatting constraints, that distracts from intent evaluation.
Based on these criteria, 15 attack methods are stratified into three difficulty levels:
- Level 1 (Basic Instruction): Direct, unambiguous harmful requests requiring no inference (e.g., standard AdvBench prompts).
- Level 2 (Simple Reasoning): Single-step modifications, such as text shuffling, ASCII art framing, or contradictory tone instructions.
- Level 3 (Complex Reasoning): Deep reasoning chains (>3 steps) or combinations of multiple Level 2 vectors (e.g., CodeAttack, multi-turn roleplay).
To test language-specific reasoning, the authors introduced several culturally and structurally grounded Level 3 attacks for Chinese:
- Chinese Character Disassembly Attack: Decomposing Chinese characters into individual sub-components and radicals to hide sensitive tokens.
- Lantern Riddle Attack: Posing the harmful query in the style of traditional riddles that imply the subject without naming it.
- Acrostic Poem Attack: Masking instructions within the first characters of poetic lines.
Datasets
- Strata-Sword Benchmark: A newly introduced bilingual (English and Chinese) hierarchical evaluation benchmark comprising 700 jailbreak instructions spanning legal violations, ethical dilemmas, hate speech, and security risks.
- Source Base Data: Built by sampling 100 core harmful queries from AdvBench, translating them into Chinese, and systematically transforming them across the 15 attack methodologies.
- Evaluation Framework: Utilized QwQ-32B as an automated judge model using evaluation taxonomies and risk criteria adapted from the Oyster safety framework.
Key Insights and Ideas
- Reasoning as an Attack Surface: As reasoning complexity increases, model defenses systematically degrade. When models dedicate compute or token budgets to decoding syntax, solving riddles, or parsing code, safety filters fail to detect the reconstructed intent.
- Cross-Lingual Asymmetry: Fine-tuning often skews toward English safety data. For example, while fine-tuning Qwen3-8B into
DS-Distill-Qwen3-8B-0528halved its English jailbreak rate (31.58% down to 16.00%), its Chinese vulnerability barely moved (26.10% down to 24.00%). - LRMs and Defensive Reasoning: Large Reasoning Models that explicitly generate Chain-of-Thought (CoT) traces (and specialized safety models like Oyster and RealSafe) show better defense ceilings, suggesting that using reasoning defensively to introspect user intent is essential to counter complex attacks.
Key Results
The authors evaluated 23 leading models across open-weight and proprietary categories:
- Escalating Failure Rates: For open-source LLMs, the average Attack Success Rate (ASR) jumped significantly across complexity tiers:
- English: Level 1 (23.83%) $\rightarrow$ Level 2 (30.64%) $\rightarrow$ Level 3 (64.42%)
- Chinese: Level 1 (28.33%) $\rightarrow$ Level 2 (40.67%) $\rightarrow$ Level 3 (70.13%)
- Commercial Models: Top proprietary models (including GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro) maintained low ASRs on Level 1 ($\le 1\%$), but their vulnerability increased on Level 3 (averaging 30.24% in English and 28.69% in Chinese).
- Most Potent Attack Vectors: Among Level 3 techniques, CodeAttack achieved the highest overall average ASR (73.24% across all models), followed by combinations like Shuffle + Template (59.60%) and the novel Character Split attack (52.69%).
Limitations and Future Work
- Multi-Turn Context Flattening: For evaluation efficiency, multi-turn attacks were concatenated into single-prompt contexts rather than conducted in real-time interactive dialogues, which may underestimate multi-turn interactive drift.
- Benchmark Scope: The benchmark focuses on 15 specific attack topologies; extending the taxonomy to multimodal inputs (e.g., vision-language jailbreaks) remains unexplored.
- Future Defense Directions: The authors emphasize that superficial alignment (pattern-matching refusals) is insufficient. Future work must investigate reasoning-driven guardrails—using CoT to deconstruct implicit intent before formulating a response.
Summary updated