Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models

Meghana Rajeev, Rajkumar Ramamurthy, Prapti Trivedi, Vikas Yadav, Oluwanifemi Bamgbose, Sathwik Tejaswi Madhusudan, James Zou, Nazneen Rajani

TL;DR

Reasoning LLMs like DeepSeek R1 and OpenAI o1 are surprisingly fragile: appending a simple, irrelevant sentence like "Interesting fact: cats sleep most of their lives" to a math problem can triple the likelihood of a model giving the wrong answer. The authors introduce CatAttack, an automated pipeline that discovers these "query-agnostic" triggers, revealing that even state-of-the-art reasoning models can be derailed by "analysis paralysis" and significant computational slowdowns.

What Problem Does This Paper Address?

Recent breakthroughs in LLMs have centered on "reasoning" models that use Chain-of-Thought (CoT) to decompose complex problems into step-by-step solutions. While these models excel at math and coding, their robustness is poorly understood.

Standard adversarial attacks usually involve changing the meaning of a prompt or finding a specific "jailbreak" for a single query. This paper investigates query-agnostic adversarial triggers: short, semantically irrelevant phrases that can be appended to any math problem to systematically mislead the model without changing the problem's logic. This poses a dual threat: it compromises the reliability of the answer and can be used as a "slowdown attack" to inflate inference costs by forcing the model to generate unnecessarily long reasoning chains.

Methods

The authors propose CatAttack, an iterative attack pipeline designed to be cost-effective and transferable.

  1. Proxy Targeting: Instead of attacking expensive, slow reasoning models (like DeepSeek R1) directly, they use a faster, cheaper "proxy" model (DeepSeek V3) to discover triggers.
  2. Iterative Optimization: An "Attacker" model (GPT-4o) generates candidate prefixes or suffixes (e.g., adding misleading hints or irrelevant trivia). A "Judge" model then checks if the proxy model failed the math problem.
  3. Semantic Filtering: To ensure the attack is truly adversarial, human annotators verify that the triggers do not actually change the math problem's meaning.
  4. Transfer: Successful triggers found on the proxy are then tested on "black-box" target models to see if the vulnerability generalizes.
The CatAttack iterative pipeline uses a proxy model to find triggers before testing them on the expensive target model.

Datasets

The study utilized several key benchmarks to discover and validate triggers:

  • Numina-math: A diverse collection of math problems used for the initial discovery of triggers.
  • GSM8K: A standard benchmark of 1,000 grade-school math word problems used to test the generalizability of the triggers across different model families.
  • CatAttack Dataset: The authors have released their discovered triggers and model responses on Hugging Face.

Key Insights and Ideas

  • Analysis Paralysis: The most effective triggers are "misleading questions" (e.g., "Could the answer possibly be around 175?"). These force the model into a self-reflection loop where it tries to reconcile the irrelevant hint with its own calculations, leading to "overthinking" and eventual failure.
  • The Distillation Gap: Distilled models (like DeepSeek-R1-Distill-Qwen) are significantly more vulnerable than their "teacher" models. The authors hypothesize that while distillation captures the output of reasoning, it fails to internalize the robustness developed by the teacher model during its Reinforcement Learning (RL) phase.
  • Susceptibility of "Easy" Problems: Counter-intuitively, triggers cause a much higher relative increase in error rates on easy problems compared to hard ones. On easy tasks, models may rely on shallow heuristic pattern-matching; when a trigger is introduced, this shallow processing is easily disrupted.
Figure 2: Relative increase in error rates after CatAttack by source. This figure shows the multiplicative increase in error rates for Deepseek-R1 and DeepSeek R1-distill-Qwen-32B models across different problem sources following the suffix attack. Sources are grouped into three difficulty categories: easy (left), medium (center), and hard (right), separated by vertical dotted lines, with no particular ordering within each category. Values for cn k12 and gsm8k for Deepseek-R1 are capped at 15x for visualization purposes, where error rates increased from 0% to non-zero levels. This capping preserves the readability of the chart while still indicating the substantial impact in these cases. We note that there were no successful errors for amc aime.

Key Results

  • Massive Error Spikes: CatAttack triggers increased the likelihood of incorrect answers by over 300% for DeepSeek R1 and up to 500-700% for other model families like Llama-3.1 and Mistral.
  • Computational Sabotage: Even when the model eventually got the right answer, the triggers caused a "slowdown." For DeepSeek R1, response lengths doubled in over 16% of cases. For the distilled Qwen model, nearly 42% of responses exceeded 1.5x their original length.
  • Cross-Family Transfer: Triggers discovered on DeepSeek models successfully transferred to Qwen QwQ, Phi-4, Llama-3.1, and Mistral, proving the vulnerability is a fundamental characteristic of current LLM reasoning architectures.
Figure 3: Response length analysis before and after CatAttack. Scatter plots compare token lengths of responses to original prompts (x-axis) versus modified prompts (y-axis) for Deepseek-R1 (right) and DeepSeek R1-distill-Qwen-32B (left) models. Logarithmic scales are used on both axes to accommodate the wide range of response lengths. Diagonal lines represent different multipliers of the original length (1x, 1.5x, 2x, etc.). Points above the 1x line represent responses that increased in length after modification, with higher lines indicating greater increases.

Limitations and Future Work

The authors note that while appending a specific instruction to "ignore distracting statements" helped mitigate the attack (dropping the error rate from 37.5% to 9.9% in one test), it did not solve the problem entirely. Furthermore, Supervised Fine-Tuning (SFT) on adversarial examples failed to generalize to new, unseen triggers.

Future research is needed to develop more robust Reinforcement Learning techniques that can teach models to distinguish between relevant problem constraints and irrelevant "noise" without sacrificing reasoning performance. The current study was also limited primarily to the mathematical domain; exploring how these triggers affect coding or legal reasoning remains an open question.

Summary updated

Abstract

We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers - short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem's semantics. We propose CatAttack, an automated iterative attack pipeline for generating triggers on a weaker, less expensive proxy model (DeepSeek V3) and successfully transfer them to more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distilled-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer. For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models, revealing that even state-of-the-art models remain susceptible to subtle adversarial inputs, raising security and reliability concerns. The CatAttack triggers dataset with model responses is available at https://huggingface.co/datasets/collinear-ai/cat-attack-adversarial-triggers.
BibTeX
@inproceedings{rajeev2025cats,
  author = {Meghana Rajeev and Rajkumar Ramamurthy and Prapti Trivedi and Vikas Yadav and Oluwanifemi Bamgbose and Sathwik Tejaswi Madhusudan and James Zou and Nazneen Rajani},
  title = {{Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=VrEPiN5WhM},
}
Figure