TL;DR
Modern commercial LLMs often restrict outputs strictly to text, eliminating access to logits or confidence scores that attackers traditionally use for black-box optimization. This paper shows that attackers don't need access to continuous probabilities: simply prompting the target model to compare two candidate inputs (e.g., "Which of these two inputs brings me closer to the attack objective?") provides an accurate, calibrated signal for hill-climbing optimization. Counterintuitively, because larger and more capable models are better at comparative reasoning, they are significantly more vulnerable to this attack.
What Problem Does This Paper Address?
Existing adversarial attacks on large language models (LLMs) and vision-language models (VLMs) typically require either:
- White-box access to model weights and gradients.
- Continuous output signals such as output token log-probabilities or logits to run zeroth-order optimization.
- Surrogate models to craft transfer-based adversarial examples.
However, production APIs (such as Anthropic’s Claude API) often operate in a strict text-only setting, revealing only generated text and no probability distributions. Generating adversarial inputs purely from text feedback typically relies on proxy reward models, which often fail due to distribution shift. The authors ask: Can we use the victim model’s own verbal introspection to guide adversarial optimization in a strictly text-only setting?
Methods
A naive attempt to ask a model for absolute numeric confidence (e.g., "Rate your confidence from 1 to 100") fails because LLMs heavily miscalibrate absolute numbers, collapsing to stereotypical values like 0, 5, or 95.
To overcome this, the authors introduce a preference-guided hill-climbing algorithm:
- Binary Comparative Prompting: Instead of asking for a score, the attacker presents two candidate inputs ($x_{\text{adv}}$ and a perturbed $x_{\text{new}}$) and asks the model which one best achieves the adversary's goal (e.g., "Which image looks less like a dog?" or "Which instruction is more likely to trigger an email-sending action?").
- Adversarial Hill-Climbing:
- Vision-LLMs: Uses the Square Attack framework, applying random local square perturbations ($\ell_\infty$ bounded) to image patches.
- Text Jailbreaks and Prompt Injections: Randomly resamples token slices within an appended adversarial suffix.
- If the target model indicates that $x_{\text{new}}$ is closer to the objective, the perturbation is accepted; otherwise, it is rejected, iterating until the target behavior is triggered.
- Safety Evasion via Framing: Framed as an evaluation or comparison question, queries appear harmless and frequently bypass standard safety guardrails that would otherwise trigger refusals.
Datasets
The authors evaluate their method across three distinct attack domains:
- Adversarial Examples (Vision-LLMs): A subset of 57 images from ImageNet, resized to $224 \times 224$ pixels, where target models initially had 100% clean accuracy.
- Prompt Injection (LLM Agents): A suite of 56 test scenarios adapted from AgentDojo, spanning 4 user task categories (summarization, translation, code review, resume review) across 14 injection attack types.
- Jailbreaks: 50 harmful direct requests drawn from AdvBench (curated by Chao et al., 2025), evaluated using Llama-3.1-8B and GPT-4o mini as automated harm judges.
Key Insights and Ideas
- Binary Introspection Outperforms Absolute Calibration: While models are notoriously bad at outputting absolute confidence scores, their pairwise comparative judgment is well-calibrated with ground-truth logit differences.
- The "Security Paradox": More capable models are more vulnerable. Small models (like Qwen2.5-VL-3B) fail to distinguish subtle differences between two perturbed inputs (often defaulting to the first option presented). In contrast, larger frontier models (like GPT-5 mini, Claude 3.5 Sonnet, and Llama-3.2-90B) have superior comparative reasoning, which provides high-fidelity optimization feedback directly to the attacker.
- Harmless-looking Meta-Queries Bypass Safety: On GPT-4.1 mini, this text-based method actually beat log-probability-based optimization (98% vs. 56% ASR). Direct optimization pushes for affirmative tokens (e.g., "Sure"), tripping alignment defenses, whereas asking the model to compare two bad prompts avoids triggering safety filters.
Key Results
- Adversarial Examples for VLMs:
- Standalone text-only optimization achieved 5%–50.8% Attack Success Rate (ASR) with a budget of 1,000 queries.
- Combining transfer priors with the proposed method (Transfer+ours) pushed untargeted ASR up to 94.7% on GPT-4o mini and 77.2% on Qwen2.5-VL-72B.
- On targeted attacks, GPT-5 mini’s ASR improved from 70.1% to 79.0%.
- Prompt Injections: Attack success on AgentDojo rose from baseline rates of 23.2%–42.1% up to 55.4%–87.5%, rivaling models with explicit logprob access.
- Jailbreaks: Near-perfect ASR ($\ge 98\%$) on Llama-3.1-8B/70B, GPT-4 Turbo, GPT-4.1 mini, and Claude Sonnet 3.5 within an average of just 5 to 33 queries.
Limitations and Future Work
- Model Capability Bottleneck: Less capable models (e.g., 3B parameters) lack the reasoning capability to distinguish subtle perturbations, rendering the search ineffective without stronger initializations.
- Strict Alignment / Rejection: Strongly aligned models can sometimes detect the comparison task as malicious and refuse to answer (e.g., GPT-4o mini frequently refused to compare jailbreak candidates).
- Future Work & Defenses:
- Expanding safety alignment to teach models to refuse comparative feedback on sensitive inputs.
- Stateful detection to identify repetitive optimization queries issued over APIs.
- Exploring multi-prompt ensembles, majority voting, or universal suffix transfers to boost query efficiency.
Summary updated