Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System

Julian Collado, Kevin Stangl

TL;DR

This paper introduces "Keep on Swimming" (KoS), a novel adversarial attack method for multi-model AI systems. KoS allows attackers to compromise the entire system even with only partial knowledge (a proxy model for the final component), overcoming transformations in earlier, "black-box" components that would otherwise neutralize attacks. This significantly improves attack success rates compared to existing methods.

What Problem Does This Paper Address?

Modern AI systems are increasingly built by composing multiple models or "agents" to tackle complex tasks. This modular approach, while powerful, introduces new security vulnerabilities. Specifically, when an attacker wants to craft an adversarial attack against such a system, they often face a significant challenge: they might not have access to the internal workings (weights) of all components. It's often infeasible to train a complete "proxy model" that mimics the entire system or even individual early-stage components. This is especially true when early components perform complex transformations (like image segmentation or text processing) that can effectively "wash away" adversarial perturbations before they reach the final, more accessible component. Existing adversarial attack methods struggle in this scenario, as they typically rely on either full model knowledge or a proxy for the entire system.

Methods

The core innovation is the "Keep on Swimming" (KoS) attack algorithm. KoS is designed for a threat model where an attacker has black-box access to the initial model(s) ($h_1$) and white-box or proxy access to the final model ($h_2$). The key challenge is that perturbations crafted for $h_2$ might be rendered ineffective by $h_1$'s transformation.

KoS addresses this iteratively:

  1. Initial Attack: An adversarial perturbation is crafted for the final model ($h_2$) using its proxy.
  2. Re-insertion and Transformation: This adversarial perturbation is "re-inserted" into the original input and passed through the initial model ($h_1$).
  3. Check for Survival: The algorithm checks if the adversarial perturbation has survived $h_1$'s transformation. This is done by verifying if the output of $h_1$ (after the perturbation) is still in the expected domain or format that $h_2$ can process effectively.
  4. "Keep on Swimming":
    • If the perturbation is still effective and the target output is achieved, the attack is successful.
  5. If the perturbation is still effective but the target is not yet reached, the attack continues on $h_2$ for more iterations.
  6. If $h_1$'s transformation has altered the input (i.e., the adversarial perturbation was partially or fully neutralized), the algorithm "keeps on swimming" by taking the new output of $h_1$ as the basis for the next adversarial attack on $h_2$. This effectively adapts the attack to the transformed input.

This iterative process allows the attack to adapt and persist through the transformations of the non-proxy-able components.

Datasets

The paper uses a dataset of images of checks for their experiments. The specific dataset is described as "pictures of checks filled with handwritten information." The goal is to misread the numerical amount on the check.

Key Insights and Ideas

  • Partial Knowledge is Sufficient: The most significant insight is that attackers don't need full knowledge of every component in a multi-model system to launch effective end-to-end attacks. Partial knowledge, specifically a proxy for the final stage, can be enough.
  • Transformations as a Defense Weakness: Instead of being a robust defense, the complex transformations in early model components can be a vulnerability. If an attacker can adapt their attack to these transformations, they can bypass defenses that rely on them.
  • Iterative Adaptation is Key: The "Keep on Swimming" approach highlights the importance of iterative adaptation. The attack doesn't just try once; it continuously re-evaluates and re-adapts based on how the intermediate components modify the adversarial perturbation.
  • Real-World Threat Model: The paper argues that their threat model (partial proxy access) is more realistic for many production AI systems, especially those with proprietary or closed-source components.

Key Results

The KoS attack demonstrated significantly better performance compared to baseline methods on their check image OCR task:

  • Success Rate: KoS achieved an 80% attack success rate, a substantial improvement over the 25% success rate of the "Crop Robust ISDP" baseline and the mere 5% of the "ISDP Baseline."
  • Perturbation Size (MSE): KoS resulted in 9.4% smaller Mean Squared Error (MSE) perturbations compared to prior state-of-the-art methods, indicating less visually noticeable changes.
  • Levenshtein Distance: The KoS pipeline also achieved lower Levenshtein distances for the final output compared to the baselines, meaning the misread numerical output was closer to the target.
  • Time: KoS introduced more noise and took more time than the simpler ISDP baseline but less time than the more complex Crop Robust ISDP. The authors emphasize that the significantly higher success rate Pareto dominates these trade-offs.

Limitations and Future Work

  • Generalization: While the authors are confident their attack will generalize to other multi-model settings (e.g., LLM pipelines, agentic systems), their experiments were focused on a supervised image pipeline.
  • Convergence Properties: The paper mentions that further work is needed to study the convergence properties of KoS.
  • Query Complexity: Future work will characterize the query complexity of KoS, though they expect it to be lower than creating a full proxy model.
  • LLM Composition: Generalizing the attack to compositions of Large Language Models (LLMs) is a key future direction.
  • Human Perception: The paper notes that while their attack is successful in fooling the models, the "HopSkipJumpAttack" (Appendix A.1) produced adversarial samples that were too visually evident for human viewers, suggesting a need for attacks that balance model deception with human imperceptibility.

Summary updated

Abstract

Recent approaches in machine learning often solve a task using a composition of multiple models or agentic architectures. When targeting a composed system with adversarial attacks, it might not be computationally or informationally feasible to train an end-to-end proxy model or a proxy model for every component of the system. We introduce a method to craft an adversarial attack against the overall multi-model system when we only have a proxy model for the final black-box model, and when the transformation applied by the initial models can make the adversarial perturbations ineffective. Current methods handle this by applying many copies of the first model/transformation to an input and then re-use a standard adversarial attack by averaging gradients, or learning a proxy model for both stages. To our knowledge, this is the first attack specifically designed for this threat model and our method has a substantially higher attack success rate (80\% vs 25\%) and contains 9.4\% smaller perturbations (MSE) compared to prior state-of-the-art methods. Our experiments focus on a supervised image pipeline, but we are confident the attack will generalize to other multi-model settings [e.g. a mix of open/closed source foundation models], or agentic systems.
BibTeX
@inproceedings{collado2024keep,
  author = {Julian Collado and Kevin Stangl},
  title = {{Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System}},
  booktitle = {LXAI @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=B8qgEE7vpQ},
}
Figure