Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions

Rachneet Sachdeva, Rima Hazra, Iryna GurevycharXiv 2025 · DOI · arXiv

Introduction

Despite rigorous safety alignment techniques like Reinforcement Learning from Human Feedback (RLHF), Large Language Models (LLMs) remain susceptible to "jailbreaks"—adversarial prompts designed to bypass ethical guardrails. While many existing attacks rely on brute-force optimization or role-playing, they often leave a clear trail of malicious intent that modern safety filters can detect. This research identifies a more subtle vulnerability: reasoning-driven exploits. The study introduces POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), a framework that turns a model's ability to follow complex instructions and provide contrastive reasoning into a weakness.

Methodology: The POATE Framework

POATE operates by disguising malicious requests as pedagogical or safety-oriented comparisons. The framework consists of two primary stages:

  1. Stage I: Polar Opposite Generation: Using a few-shot approach, the attack transforms a harmful query (e.g., "How to poison food") into its "safe" semantic opposite ("How to ensure food safety").
  2. Stage II: Adversarial Template Construction & Elaboration: The attack crafts a prompt asking the model to provide a tutorial on the safe task while insisting it explain the "opposing process" in detail to help the reader understand what to avoid. If the model refuses due to specific keywords, the framework automatically generates a version with "implicit intent," removing the keyword but keeping the logic intact.
Overview of the POATE framework, showing the transformation of a harmful query into a polar opposite and the subsequent adversarial template construction.

Experimental Evaluation

The study evaluates POATE across three prominent benchmarks: AdvBench (standard harmful instructions), XSTest (exaggerated safety checks), and MaliciousInstructions (diverse harmful categories). The models tested include various sizes of Llama, Gemma, Phi, and GPT-3.5/4o.

The Intelligence Paradox

A striking finding is the "Intelligence Paradox": larger, more capable models are often more vulnerable to POATE. Superior instruction-following and reasoning capabilities make these models more likely to follow the "logic" of a contrastive prompt into a safety violation. For instance, GPT-4o and Llama-3.1-70B consistently showed higher Attack Success Rates (ASR) than their smaller counterparts.

Attack Success Rate (%) across different models and datasets, illustrating that larger models (e.g., GPT-4o, Llama-3.1-70B) often exhibit higher vulnerability.

Key Results and Insights

  • High Success Rate: POATE achieved an average ASR of approximately 57% across all datasets and models, significantly outperforming established baselines like GCG (~22%) and DeepInception (~20%).
  • Category Vulnerability: Models were most susceptible to Fraud/Deception (71.6% ASR) and Hate/Harassment (66% ASR), while remaining more robust against Physical Harm (21.5% ASR).
  • Semantic Cloaking: By framing requests as "prevention," the attack bypasses surface-level intent filters. The model perceives the primary goal as safe and treats the generation of harmful content as a secondary educational requirement.

Proposed Defenses

To counter reasoning-driven exploits, the authors propose two Chain-of-Thought (CoT) strategies:

  • Intent-Aware CoT: Forces the model to explicitly decompose the query and identify hidden malicious intents before answering.
  • Reverse Thinking CoT: Instructs the model to imagine how its potential response could be misused, regardless of the prompt's phrasing. This was the most effective defense, reducing the POATE success rate by an average of 95% with minimal latency overhead.

Conclusion and Future Work

POATE demonstrates that the very reasoning capabilities that make LLMs powerful can be leveraged for deception. While inference-time CoT defenses are highly effective, future research should explore mitigating these vulnerabilities during the training and alignment phases (e.g., via SFT or DPO). Additionally, the potential for multi-turn conversational jailbreaks remains an open and critical area for investigation.

Summary updated

Abstract

Large language models, despite extensive alignment with human values and ethical principles, remain vulnerable to sophisticated jailbreak attacks that exploit their reasoning abilities. Existing safety measures often detect overt malicious intent but fail to address subtle, reasoning-driven vulnerabilities. In this work, we introduce POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses. POATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety. We conduct extensive evaluation across six diverse language model families of varying parameter sizes to demonstrate the robustness of the attack, achieving significantly higher attack success rates (~44%) compared to existing methods. To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses. These methods enhance reasoning robustness and strengthen the model's defense against adversarial exploits.
BibTeX
@misc{sachdeva2025turning,
  author = {Rachneet Sachdeva and Rima Hazra and Iryna Gurevych},
  title = {{Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions}},
  year = {2025},
  eprint = {2501.01872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01872},
}
Figure