Black-Box Adversarial Attacks

383 concepts351 papers3 proposals

White-box adversarial attacks are conceptually straightforward: if you have full access to a model's weights, you can compute the exact gradient of the loss function with respect to the input. Moving the input a tiny step along this gradient direction directly maximizes the model's error.

However, in real-world deployments, machine learning models are typically exposed as black boxes. Whether it is a proprietary large language model API, a cloud-based computer vision service, or an embedded controller in an autonomous vehicle, the internal parameters, gradients, and training data are hidden from the user. As highlighted by Zheng et al. in their comprehensive benchmark, this lack of transparency introduces a fundamental optimization barrier: how can an adversary find a minimal, deceptive perturbation when they cannot backpropagate through the target model?

To overcome this constraint, researchers have developed strategies that fall into three primary paradigms, as categorized by Mahmood et al.: score-based attacks, decision-based attacks, and transfer-based attacks.

Score-Based Attacks and Query-Efficient Optimization

When a black-box model provides continuous outputs—such as class probabilities, confidence scores, or regression values—attackers can treat the system as a black-box function to be optimized via external queries.

A foundational approach to this is SimBA (Simple Black-box Attack), proposed by Chuan Guo. Instead of attempting to explicitly reconstruct the full gradient vector, SimBA iteratively samples random orthogonal vectors and either adds or subtracts them from the input image. By observing whether the model's confidence score for the correct class decreases, the attack greedily paths toward an adversarial example.

While effective, direct random search can require thousands of queries, making it easily detectable by security firewalls. To improve query efficiency, researchers inject structural priors into the search space. For instance, Liu et al. introduce the Spatial Transform Black-box Attack (STBA), which restricts perturbations to high-frequency regions of an image, achieving high success rates with far fewer queries. In the audio domain, Kamsetty et al. combine evolutionary genetic algorithms with query-based local gradient estimation to fool automatic speech recognition systems.

Alternatively, attackers can optimize within a lower-dimensional latent space rather than the raw input space. Dolatabadi et al. propose AdvFlow, which utilizes Normalizing Flows to generate perturbations that adhere strictly to the natural data distribution, rendering the resulting adversarial examples highly inconspicuous. For complex red-teaming scenarios, Lee et al. leverage Bayesian Optimization to systematically map out model vulnerabilities within a strict query budget.

Decision-Based Attacks and Discrete Optimization

The challenge intensifies when a model provides no confidence scores at all, returning only a final discrete output, such as a hard label, a text token, or a ranked list. In these decision-based settings, gradient estimation via local finite differences fails because small input changes yield zero change in the output label until a decision boundary is crossed.

To navigate this, attackers rely on discrete optimization and combinatorial search. For example, Torcinovich et al. deploy specialized Genetic Algorithms to inject data poison into clustering algorithms where only final cluster assignments are visible. Similarly, Chen et al. utilize frequency-domain decision attacks to bypass face forgery detectors using only binary outputs.

Discrete domains like natural language and source code present an even steeper hurdle because inputs cannot be infinitesimally perturbed. In text-guided image generation, Pei et al. treat characters as pixels to optimize prompt perturbations. For large language models (LLMs), Xue et al. introduce TrojLLM, a framework that uses reinforcement learning to discover universal discrete trigger prompts that activate hidden Trojans without needing internal weight access. In the context of code completion engines, Jenko et al. demonstrate that injecting optimized comments can trick proprietary models into suggesting insecure code. When targeting information retrieval, Ruqing Zhang develops an RL framework called RELEVANT that manipulates document rankings across whole topic groups based solely on the observed final rank positions.

Transfer-Based Attacks and Surrogate Models

When a target system completely blocks repetitive querying or hides behind a strict rate limit, query-based optimization becomes impossible. Attackers instead exploit a fascinating property of deep neural networks: adversarial transferability. Adversarial examples crafted to fool one model often successfully fool an entirely different, unseen model trained on the same task.

In a transfer-based attack, the adversary trains a local "surrogate" or "substitute" model to mimic the target. They then compute white-box gradients against this surrogate to generate adversarial inputs. As Yaoyao Zhong demonstrates in deep face recognition, these locally generated examples can bypass target systems without requiring a single query of feedback.

However, standard white-box attacks often "overfit" to the specific decision boundaries and local loss spikes of the surrogate model, causing the transfer rate to drop. To resolve this, Fan et al. introduce Reverse Adversarial Perturbation (RAP).

Standard Attack Optimization:
Surrogate Loss Minimum ──> (Often a narrow, sharp valley unique to the surrogate)

RAP Optimization:
Surrogate Loss Flat Region ──> (Generalized features shared across different architectures)

RAP explicitly guides the optimization toward "flat" regions of the surrogate's loss landscape. Because flat minima represent more generalized features of the underlying data distribution, perturbations found in these regions are significantly more robust and transfer effectively to unknown commercial APIs. In recommendation systems, Zhang et al. implement a local ensemble simulator combined with influence functions to accurately estimate how injected user profiles will transfer to manipulate an opaque next-item recommendation engine.

Expanding to Complex and Multi-Agent Environments

As machine learning expands into sequential decision-making and multi-agent systems, black-box attacks have evolved beyond static inputs. In deep reinforcement learning for continuous control, Weng et al. show that an attacker can degrade an agent's reward by learning a model of the environment's dynamics purely from observation, systematically steering the agent into unsafe states. In collaborative environments, Wu et al. show that multi-agent robot teams can be hijacked by intercepting and perturbing their communication channels, bypassing the need to know individual neural network weights or mission objectives.

Even recommendation platforms are vulnerable to profile-copying strategies. Instead of generating synthetic noise, Fan et al. introduce CopyAttack, which uses reinforcement learning to select and inject authentic user profiles copied from a completely different domain. This leverages real user behavioral distributions to manipulate recommendation lists while effortlessly evading traditional fraud detection systems.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_black_box_adversarial_attacks_ee567109,
  title = {black-box adversarial attacks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-adversarial-attacks/rdr_ee5671091f0c4624b81c7ad6f0e7bef2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_black_box_adversarial_attacks_ee567109_v1,
  title = {black-box adversarial attacks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-adversarial-attacks/rdr_ee5671091f0c4624b81c7ad6f0e7bef2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake