Gradient Based Discrete Optimization for Adversarial Prompts

This direction is split into other directions: white-box optimization dependency, gradient-based attack constraints. Its last version stays readable and citable here.

Version 1 of 1502 papers517 conceptsfrozen 2026-10-04

In the world of continuous neural networks, text is an anomaly. While we can easily nudge the pixels of an image to change a model's classification, we cannot "nudge" a word. You cannot move 0.01 units from the word "apple" toward the word "orange." This discrete nature of language makes optimizing prompts—whether for jailbreaking, data extraction, or debugging—a significant challenge.

Researchers have bridged this gap by developing techniques that use a model's internal gradients to guide the search for discrete tokens. By treating the model's continuous embedding space as a map, they can identify which "directions" lead toward a desired (or undesired) behavior and then find the specific words that sit in those locations.

The Gradient to Token Bridge

The foundational challenge of adversarial prompt optimization is the "discrete bottleneck." Standard backpropagation tells us how to change a continuous value to reduce loss, but it doesn't tell us which token to pick from a vocabulary of 50,000 options.

Yuxin Wen and colleagues addressed this in Hard Prompts Made Easy (PEZ) by optimizing in the continuous embedding space and then projecting those embeddings back onto the nearest valid tokens. This approach allows for the discovery of "hard" prompts—actual strings of text—that can bypass filters or trigger specific model behaviors.

Building on this, the Greedy Coordinate Gradient (GCG) method became a standard for jailbreaking. However, GCG is computationally expensive. Yiran Zhao introduced Probe Sampling to accelerate this process. By using a smaller "draft" model to filter out unpromising token candidates before verifying them on the target model, Probe Sampling achieves up to a 5.6x speedup, making adversarial search far more practical.

Refining the Objective

Early attacks focused simply on forcing a model to start its response with an affirmative phrase like "Sure, here is..." However, modern safety-aligned models are trained to resist this. Researchers have responded by developing more nuanced optimization objectives.

In Don't Say No (DSN), Wenjie Wang argues that it is more effective to actively suppress refusal than to merely encourage compliance. By using an "unlikelihood loss" to penalize keywords like "sorry" or "cannot," the optimization process carves out a path where the model feels it has no choice but to fulfill the harmful request. Similarly, Hao Chen demonstrated in Improved Generation of Adversarial Examples that borrowing "Skip Gradients" from computer vision can help smooth out the noisy gradients inherent in discrete text, significantly boosting attack success rates on models like Llama-2.

Optimizing Internal Representations

The most potent recent shift in this field is moving the optimization target from the output of the model to its internal states. If an attacker can manipulate the model's "thoughts," the output will naturally follow.

Ran Li introduced this concept with LARGO, which optimizes "adversarial thoughts" in the model's continuous latent space. Instead of searching for a text suffix that triggers a bad output, LARGO finds a latent representation that represents a "harmful intent" and then asks the model to translate that intent into natural language.

This is echoed in DROJ, where researchers optimize prompts to shift the model's hidden states away from "refusal" directions. By mathematically steering the internal representation of a query, they can bypass safety alignments entirely. Zihao Xu took this to the extreme with CLIP, optimizing the entire input embedding to stay within statistical bounds while remaining invisible to text-based detectors.

The White-Box Dependency and Transferability

A recurring theme across these papers is the "White-Box" requirement. To calculate gradients, an optimizer needs full access to the model's weights. This would seemingly protect proprietary models like GPT-4 or Claude, which are only accessible via API.

However, research shows a surprising degree of transferability. Prompts optimized against open-source models like Llama or Vicuna often work against closed-source models. Cen Chen showed in CCJA that by optimizing for "context-coherent" prompts—those that look like natural language—adversarial strings become even more transferable and harder for safety filters to catch.

This direction isn't limited to text. Zhi-Yi Chin (with P4D) and Fan Yang (with DO-UAP) have extended these gradient-based discrete optimization techniques to vision-language models and diffusion pipelines, proving that as long as a model uses continuous representations internally, its discrete inputs remain vulnerable to gradient-guided manipulation.

Papers in this version

Showing 60 of 502 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_based_discrete_optimization_for_c3b74e3f,
  title = {Gradient Based Discrete Optimization for Adversarial Prompts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-discrete-optimization-for-adversarial-prompts/rdr_c3b74e3fb63b4385964aa53ea67ba24c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_based_discrete_optimization_for_c3b74e3f_v1,
  title = {Gradient Based Discrete Optimization for Adversarial Prompts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-discrete-optimization-for-adversarial-prompts/rdr_c3b74e3fb63b4385964aa53ea67ba24c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}