TL;DR
Reasoning LLMs like DeepSeek R1 and OpenAI o1 are surprisingly fragile: appending a simple, irrelevant sentence like "Interesting fact: cats sleep most of their lives" to a math problem can triple the likelihood of a model giving the wrong answer. The authors introduce CatAttack, an automated pipeline that discovers these "query-agnostic" triggers, revealing that even state-of-the-art reasoning models can be derailed by "analysis paralysis" and significant computational slowdowns.
What Problem Does This Paper Address?
Recent breakthroughs in LLMs have centered on "reasoning" models that use Chain-of-Thought (CoT) to decompose complex problems into step-by-step solutions. While these models excel at math and coding, their robustness is poorly understood.
Standard adversarial attacks usually involve changing the meaning of a prompt or finding a specific "jailbreak" for a single query. This paper investigates query-agnostic adversarial triggers: short, semantically irrelevant phrases that can be appended to any math problem to systematically mislead the model without changing the problem's logic. This poses a dual threat: it compromises the reliability of the answer and can be used as a "slowdown attack" to inflate inference costs by forcing the model to generate unnecessarily long reasoning chains.
Methods
The authors propose CatAttack, an iterative attack pipeline designed to be cost-effective and transferable.
- Proxy Targeting: Instead of attacking expensive, slow reasoning models (like DeepSeek R1) directly, they use a faster, cheaper "proxy" model (DeepSeek V3) to discover triggers.
- Iterative Optimization: An "Attacker" model (GPT-4o) generates candidate prefixes or suffixes (e.g., adding misleading hints or irrelevant trivia). A "Judge" model then checks if the proxy model failed the math problem.
- Semantic Filtering: To ensure the attack is truly adversarial, human annotators verify that the triggers do not actually change the math problem's meaning.
- Transfer: Successful triggers found on the proxy are then tested on "black-box" target models to see if the vulnerability generalizes.
Datasets
The study utilized several key benchmarks to discover and validate triggers:
- Numina-math: A diverse collection of math problems used for the initial discovery of triggers.
- GSM8K: A standard benchmark of 1,000 grade-school math word problems used to test the generalizability of the triggers across different model families.
- CatAttack Dataset: The authors have released their discovered triggers and model responses on Hugging Face.
Key Insights and Ideas
- Analysis Paralysis: The most effective triggers are "misleading questions" (e.g., "Could the answer possibly be around 175?"). These force the model into a self-reflection loop where it tries to reconcile the irrelevant hint with its own calculations, leading to "overthinking" and eventual failure.
- The Distillation Gap: Distilled models (like DeepSeek-R1-Distill-Qwen) are significantly more vulnerable than their "teacher" models. The authors hypothesize that while distillation captures the output of reasoning, it fails to internalize the robustness developed by the teacher model during its Reinforcement Learning (RL) phase.
- Susceptibility of "Easy" Problems: Counter-intuitively, triggers cause a much higher relative increase in error rates on easy problems compared to hard ones. On easy tasks, models may rely on shallow heuristic pattern-matching; when a trigger is introduced, this shallow processing is easily disrupted.
Key Results
- Massive Error Spikes: CatAttack triggers increased the likelihood of incorrect answers by over 300% for DeepSeek R1 and up to 500-700% for other model families like Llama-3.1 and Mistral.
- Computational Sabotage: Even when the model eventually got the right answer, the triggers caused a "slowdown." For DeepSeek R1, response lengths doubled in over 16% of cases. For the distilled Qwen model, nearly 42% of responses exceeded 1.5x their original length.
- Cross-Family Transfer: Triggers discovered on DeepSeek models successfully transferred to Qwen QwQ, Phi-4, Llama-3.1, and Mistral, proving the vulnerability is a fundamental characteristic of current LLM reasoning architectures.
Limitations and Future Work
The authors note that while appending a specific instruction to "ignore distracting statements" helped mitigate the attack (dropping the error rate from 37.5% to 9.9% in one test), it did not solve the problem entirely. Furthermore, Supervised Fine-Tuning (SFT) on adversarial examples failed to generalize to new, unseen triggers.
Future research is needed to develop more robust Reinforcement Learning techniques that can teach models to distinguish between relevant problem constraints and irrelevant "noise" without sacrificing reasoning performance. The current study was also limited primarily to the mathematical domain; exploring how these triggers affect coding or legal reasoning remains an open question.
Summary updated