Generating adversarial examples in a black-box setting—where the attacker only has access to a model's outputs rather than its internal weights or gradients—is notoriously expensive. Optimization algorithms must navigate high-dimensional spaces to find perturbations that alter a model's predictions while remaining imperceptible or structured.
When these attacks are initialized from scratch (a "cold start"), they typically begin with random noise or a clean input. Geometrically, this means the optimization starts deep within a region of high-confidence correct classifications. The attacker must then expend a massive number of queries simply trying to locate the model's decision boundary.
To solve this query-efficiency crisis, a powerful paradigm has emerged: warm-starting the optimization using informed initializations. Instead of searching blindly, attackers inject prior knowledge—derived from white-box surrogate models, historical attack data, or structural properties of the target domain—to seed the optimization process close to or directly on the decision boundary.
Transfer-Based Priors from Surrogate Models
The most common way to warm-start a black-box attack is to leverage the concept of adversarial transferability. Because different machine learning models often learn similar decision representations, an adversarial perturbation that fools a local, fully accessible "surrogate" model will often reside close to the decision boundary of the target black-box model.
Cold Start: [Clean Image] ------(Many Queries in the Dark)------> [Decision Boundary]
Warm Start: [Clean Image] -> [Surrogate White-Box Attack] -> [Boundary Proximity] --(Few Queries)--> [Target Boundary]
Even when a transfer attack fails to outright fool the target model, it provides an exceptional starting point. As demonstrated by Fnu Suya et al., these failed transfer candidates are not useless; they typically sit significantly closer to the target's decision boundary than a random starting point, reducing the remaining optimization cost by over 75%.
This geometric advantage is particularly pronounced in hard-label or ray-search attacks, where the attacker only receives the final predicted class label. Chen Ma et al. showed that initializing a black-box ray search with a starting point derived from a white-box surrogate attack drastically minimizes the initial perturbation distance, bypassing the most query-intensive phase of the boundary search.
Furthermore, this shared adversarial subspace exists even under highly restrictive conditions. Nathan Inkawhich demonstrated that transfer-based noise can successfully warm-start query-based attacks even when the source and target models share zero training data and have completely non-overlapping label spaces. Similarly, in specialized domains like face forgery detection, Zhaoyu Chen et al. bootstrapped black-box attacks by exploiting the correlation between the intermediate features of face recognition models and forgery detectors, avoiding the image degradation caused by cold-starting with random noise.
Without this transfer-based initialization, evolutionary and gradient-free optimization methods suffer from severe inefficiencies. As noted by Yu Zhan et al. in the context of video classification, starting from scratch introduces a severe cold-start problem that exponentially inflates the query budget.
Structural and Data-Driven Initializations
When a reliable surrogate model is unavailable, attackers can instead warm-start optimization by exploiting the structural properties of the input data, the target task, or historical attack patterns.
Target-to-Source Inversion
In decision-based attacks, the goal is often to find the smallest possible perturbation that changes a source image into a target classification. Pu Zhao et al. flipped the traditional optimization direction: they initialized the attack with an image genuinely belonging to the target class and iteratively shrank it toward the source image. This ensures the optimization always operates along the decision boundary, drastically reducing query overhead.
Neutral and Pre-Distorted Baselines
Conversely, the features of the source image itself can sometimes act as an obstacle for the optimizer. To circumvent this, Yali Du et al. proposed initializing attacks from a neutral gray image. This provides a "blank slate" that prevents the optimizer from having to waste queries overcoming existing visual features to reach a target classification.
In other scenarios, the target system relies on a specific signal that can be degraded before optimization begins. For instance, when evading AI image watermarks, Jinghuai Zhang et al. initialized their search using a JPEG-compressed version of the image. Because compression naturally degrades the watermark signal, this informed starting point allows the optimizer to find high-fidelity, imperceptible evasions with far fewer queries than a random initialization.
Recycling and Subspace Projections
Adversarial vulnerabilities often cluster within low-dimensional subspaces. Rather than engineering a new perturbation for every target, attackers can recycle structural patterns that worked previously:
- Dictionary Retrieval: Junyoung Byun et al. developed a dictionary-based system that stores successful adversarial patterns from past attacks. When targeting a new face recognition subject, the system retrieves and adapts a historically successful noise pattern, bypassing the early stages of optimization.
- Principal Component Initialization: Jens Bayer et al. applied Principal Component Analysis (PCA) to a bank of successful adversarial patches to isolate "Eigenpatches." Initializing an attack using a combination of these pre-computed basis vectors allows the optimization to start directly within a high-potential adversarial subspace.
- Generative Averaging: For reconstruction and privacy-breaching attacks, Hui Zhang et al. showed that averaging high-performing samples from a generative model creates an optimized initial identity anchor, making practical face reconstruction possible within a limited query budget.
Extending Warm-Starts to Discrete and Multimodal Spaces
The intuition behind warm-starting is not confined to continuous image pixels; it translates directly to discrete tokens in Large Language Models (LLMs) and multimodal vision-language systems.
In discrete spaces, optimization is exceptionally difficult because gradients are not naturally available. To jailbreak LLMs via reinforcement learning, Piyush Jha et al. seeded the optimization process with a library of previously successful adversarial suffixes. This places the RL agent immediately inside high-potential regions of the prompt space, accelerating convergence toward novel jailbreaks.
Discrete Token Space:
[Random Suffix] --------(Combinatorial Explosion / Failed Search)--------> Low Success
[Successful Suffix Prior] -> [RL Fine-Tuning] -> [Rapid Convergence] -> High Success
For vision-language models, attackers combine visual and textual priors to bypass safety filters. Ruoxi Cheng et al. demonstrated that pre-conditioning a black-box jailbreak with an adversarially generated image provides a powerful multi-modal prior that maximizes toxicity generation. When optimizing universal patches against these models without internal access, Daizong Liu et al. utilized a semantic-driven initialization strategy: they sampled random patch configurations and selected the one that yielded the smallest initial semantic distance to the target malicious response.
Finally, Haoxuan Ji et al. revealed that initializing optimization with inputs semantically related to the target query primes the model's internal attention mechanisms. This structural priming lowers the model's resistance, making it significantly easier for subsequent optimization steps to force the system into an adversarial state.
Key Foundational Work
Hybrid Batch Attacks: Finding Black-box Adversarial Examples with Limited Queries (2019) — contributed 2 concept elements to this direction.
On the Design of Black-box Adversarial Examples by Leveraging Gradient-free Optimization and Operator Splitting Method (2019) — contributed 1 concept elements to this direction.
Towards Query Efficient Black-box Attacks: An Input-free Perspective (2018) — contributed 1 concept elements to this direction.