Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions

Shiji Zhao, Ranjie Duan, Jiexi Liu, Xiaojun Jia, Fengxiang Wang, Ruoxi Cheng, Yong xie, Cheng Wei, Yitong Sun, Chang Liu, Qing Guo, Jialing Tao, Hui Xue, Xingxing WeiSubmission to ICLR 2026 · Rejected

TL;DR

Strata-Sword introduces a hierarchical safety benchmark that evaluates Large Language Models (LLMs) and Large Reasoning Models (LRMs) against jailbreak attacks categorized by their underlying reasoning complexity. By testing 23 frontier and open-weight models across English and Chinese, the authors demonstrate that while current models reliably refuse basic harmful prompts, their safety guardrails frequently collapse when harmful intent is obscured behind multi-step logic, linguistic obfuscation, or cultural puzzles.

What Problem Does This Paper Address?

Most safety alignment techniques (such as RLHF or DPO) train models to detect and reject explicit, surface-level harmful requests (e.g., direct queries asking for illicit instructions). However, as models have become stronger at multi-step reasoning, this capability has become a double-edged sword: adversaries can disguise malicious intentions within complex instructions that require semantic deconstruction or code execution to uncover.

Existing safety benchmarks typically measure the breadth of harmful topics or push the brute-force strength of attack prompts, but they rarely measure at what level of cognitive reasoning a model's safety guardrails fail. Furthermore, safety evaluations are heavily skewed toward English, often ignoring how language-specific structural features—such as the logographic and semantic nature of Chinese characters—can be leveraged to bypass safety filters.

Methods

The authors quantify Reasoning Complexity along three foundational dimensions:

  1. Logical Depth: The number of multi-step, causal, or conditional inferences required to infer the core intent.
  2. Linguistic Ambiguity: The use of syntactic permutations, figurative language, or semantic obfuscation.
  3. Task Overhead: Extraneous cognitive load, such as code analysis or formatting constraints, that distracts from intent evaluation.

Based on these criteria, 15 attack methods are stratified into three difficulty levels:

  • Level 1 (Basic Instruction): Direct, unambiguous harmful requests requiring no inference (e.g., standard AdvBench prompts).
  • Level 2 (Simple Reasoning): Single-step modifications, such as text shuffling, ASCII art framing, or contradictory tone instructions.
  • Level 3 (Complex Reasoning): Deep reasoning chains (>3 steps) or combinations of multiple Level 2 vectors (e.g., CodeAttack, multi-turn roleplay).

To test language-specific reasoning, the authors introduced several culturally and structurally grounded Level 3 attacks for Chinese:

  • Chinese Character Disassembly Attack: Decomposing Chinese characters into individual sub-components and radicals to hide sensitive tokens.
  • Lantern Riddle Attack: Posing the harmful query in the style of traditional riddles that imply the subject without naming it.
  • Acrostic Poem Attack: Masking instructions within the first characters of poetic lines.

Datasets

  • Strata-Sword Benchmark: A newly introduced bilingual (English and Chinese) hierarchical evaluation benchmark comprising 700 jailbreak instructions spanning legal violations, ethical dilemmas, hate speech, and security risks.
  • Source Base Data: Built by sampling 100 core harmful queries from AdvBench, translating them into Chinese, and systematically transforming them across the 15 attack methodologies.
  • Evaluation Framework: Utilized QwQ-32B as an automated judge model using evaluation taxonomies and risk criteria adapted from the Oyster safety framework.

Key Insights and Ideas

  • Reasoning as an Attack Surface: As reasoning complexity increases, model defenses systematically degrade. When models dedicate compute or token budgets to decoding syntax, solving riddles, or parsing code, safety filters fail to detect the reconstructed intent.
  • Cross-Lingual Asymmetry: Fine-tuning often skews toward English safety data. For example, while fine-tuning Qwen3-8B into DS-Distill-Qwen3-8B-0528 halved its English jailbreak rate (31.58% down to 16.00%), its Chinese vulnerability barely moved (26.10% down to 24.00%).
  • LRMs and Defensive Reasoning: Large Reasoning Models that explicitly generate Chain-of-Thought (CoT) traces (and specialized safety models like Oyster and RealSafe) show better defense ceilings, suggesting that using reasoning defensively to introspect user intent is essential to counter complex attacks.

Key Results

The authors evaluated 23 leading models across open-weight and proprietary categories:

  • Escalating Failure Rates: For open-source LLMs, the average Attack Success Rate (ASR) jumped significantly across complexity tiers:
  • English: Level 1 (23.83%) $\rightarrow$ Level 2 (30.64%) $\rightarrow$ Level 3 (64.42%)
  • Chinese: Level 1 (28.33%) $\rightarrow$ Level 2 (40.67%) $\rightarrow$ Level 3 (70.13%)
  • Commercial Models: Top proprietary models (including GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro) maintained low ASRs on Level 1 ($\le 1\%$), but their vulnerability increased on Level 3 (averaging 30.24% in English and 28.69% in Chinese).
  • Most Potent Attack Vectors: Among Level 3 techniques, CodeAttack achieved the highest overall average ASR (73.24% across all models), followed by combinations like Shuffle + Template (59.60%) and the novel Character Split attack (52.69%).

Limitations and Future Work

  • Multi-Turn Context Flattening: For evaluation efficiency, multi-turn attacks were concatenated into single-prompt contexts rather than conducted in real-time interactive dialogues, which may underestimate multi-turn interactive drift.
  • Benchmark Scope: The benchmark focuses on 15 specific attack topologies; extending the taxonomy to multimodal inputs (e.g., vision-language jailbreaks) remains unexplored.
  • Future Defense Directions: The authors emphasize that superficial alignment (pattern-matching refusals) is insufficient. Future work must investigate reasoning-driven guardrails—using CoT to deconstruct implicit intent before formulating a response.

Summary updated

Abstract

Large language models (LLMs) have gained widespread recognition for their superior performance and have been deployed across numerous domains. Building on Chain-of-Thought (CoT) ideology, Large Reasoning models (LRMs) further exhibit strong reasoning skills, enabling them to infer more accurately and respond appropriately. However, strong general reasoning capabilities do not guarantee a safety response to jailbreak instructions requiring even more robust reasoning capabilities. A model with strong general reasoning capabilities but lacking corresponding safety capabilities can create serious vulnerabilities in the real application. Therefore, a comprehensive benchmark needs to be established to evaluate the safety performance of the model in the face of instructions of different reasoning complexity, which can provide a new dimension of the safety boundaries of the LLMs. This paper quantifies "Reasoning Complexity" as an evaluable safety dimension and categorizes 15 jailbreak attack methods into three different levels according to the reasoning complexity, establishing a hierarchical Chinese-English jailbreak safety benchmark for systematically evaluating the safety performance of LLMs. Meanwhile, to fully consider reasoning complexity brought by unique language characteristics, we first propose some Chinese jailbreak attack methods, including the Chinese Character Disassembly attack, Lantern Riddle attack, and Acrostic Poem attack. A series of experiments indicate that current LLMs and LRMs show different safety boundaries under different reasoning complexity, which provides a new perspective to develop safer LLMs and LRMs
BibTeX
@misc{zhao2025strata,
  author = {Shiji Zhao and Ranjie Duan and Jiexi Liu and Xiaojun Jia and Fengxiang Wang and Ruoxi Cheng and Yong xie and Cheng Wei and Yitong Sun and Chang Liu and Qing Guo and Jialing Tao and Hui Xue and Xingxing Wei},
  title = {{Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions}},
  year = {2025},
  eprint = {2509.01444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.01444},
}
Figure