In the "cat-and-mouse" game of AI security, the most significant bottleneck for an attacker is often the "cold start." Whether attempting to jailbreak a Large Language Model (LLM) or fool a computer vision system, starting from random noise or arbitrary strings requires a massive number of queries to find a model's vulnerabilities.
Recent research demonstrates that the efficiency of an adversarial attack depends less on the complexity of the optimization algorithm and more on the quality of the starting line. By using informed initialization—seeding the search with pre-optimized patterns, surrogate model outputs, or semantic priors—attackers can reduce the computational cost of finding exploits by over 90%.
The Cold Start Problem in Adversarial Search
Most adversarial attacks are framed as optimization problems: find a small perturbation that maximizes the model's error. In a "black-box" setting, where the attacker cannot see the model's internal gradients, they must "poke" the model repeatedly to estimate the direction of the decision boundary.
Starting with random initialization is like trying to find a needle in a haystack while blindfolded. As noted by Thomas Brunner and colleagues in Copy and Paste, the initial query budget is often exhausted just trying to find any adversarial sample before the actual optimization even begins. This "initialization cost" makes many real-world systems appear more robust than they actually are.
Transfer-Based Seeding: Leveraging Surrogate Models
The most common strategy to bypass the cold start is to use a "proxy" or surrogate model. If two models are trained on similar data, they often share similar decision boundaries.
Researchers like Fnu Suya in Hybrid Batch Attacks and Tao Xiang in Local Black-box Adversarial Attacks have shown that even if a white-box attack fails to fool the target model directly, the resulting candidate is usually much closer to the target's decision boundary than a random point. This "transfer-based seeding" provides a high-quality initial direction, allowing subsequent optimization to converge with orders of magnitude fewer queries.
This principle extends to complex domains like face recognition. Xiao Yang and team demonstrated in Adversarial Textured 3D Meshes that initializing an attack with the victim's facial coefficients—rather than random mesh perturbations—aligns the search with the target's identity manifold from the very first step.
Semantic and Structural Priors
Beyond surrogate models, attackers can initialize searches using "priors"—knowledge about how models are built or how they typically fail.
- Sanitizer Inversion: In the context of code completion, Slobodan Jenko and Martin Vechev developed INSEC (Black-Box Adversarial Attacks on LLM-Based Code Completion). They initialize attacks by injecting comments that suggest security checks have already been performed, effectively "priming" the model to suggest insecure code.
- Salient Feature Transplantation: In Copy and Paste, researchers found that simply pasting a small patch of a target class (e.g., a lion's eyes) onto a source image (a water bottle) creates a starting point that is already misclassified. This reduces the required queries by 81% because the optimizer only needs to "polish" the noise rather than discover the target class from scratch.
- Task Correlation: Bo Li showed in Exploring Decision-based Black-box Attacks on Face Forgery Detection that perturbations from a related task (identity verification) can serve as a "warm start" for attacking deepfake detectors, exploiting the shared feature representations between the two tasks.
Seeding the "Compliance Direction" in LLMs
For Large Language Models, the search space is discrete (tokens), making optimization particularly difficult. However, jailbreak attacks often follow predictable patterns.
Amit Levi and colleagues introduced CRI (Compliance Refusal Initialization) (Enhancing Jailbreak Attacks via Compliance-Refusal-Based Initialization), which posits that models have an internal "compliance direction." By selecting starting points from a library of successful suffixes that already move the model's activations toward this direction, they achieved 100% success rates while reducing computational costs by 98%.
Similarly, Hanqing Liu and Lifeng Zhou in SI-GCG and Tianyu Pang in I-GCG found that adversarial suffixes are highly transferable. A suffix that successfully jailbreaks a "simple" harmful request can serve as a powerful seed for a more difficult, robustly aligned prompt.
Geometric Initialization in Representation Space
A more theoretical approach involves finding the "nearest" decision boundary in the model's internal representation space. In Rethinking Classifier and Adversarial Attack, researchers proposed Absolute Classification Boundaries Initialization (ACBI). Instead of starting with random noise, they use gradient descent to shift an input toward the nearest linear boundary in the representation space. This targeted initialization consistently reveals that "robust" models are more vulnerable than previously thought, as it avoids the local minima that plague random starts.
Why It Works: Navigating the Landscape
The intuition behind the success of these methods lies in the geometry of the loss landscape. Random initialization often places the starting point in a "flat" region where the model is highly confident in its correct prediction. Informed initialization moves the starting point onto the "slopes" or near the "cliffs" (the decision boundaries).
By starting closer to the point of failure, optimization algorithms like the Genetic Algorithms used in QuScore or the Reinforcement Learning in LLM STINGER can focus their limited query budget on refinement rather than discovery. As Ashkan Esmaeili demonstrates in LSDAT, maintaining a hierarchical library of these "universal starting points" allows new attacks to bypass the most expensive phase of the adversarial process entirely.