Evaluating or training a neural network against adversarial inputs requires finding the "worst-case" input perturbation that maximizes the model's loss. In computer vision, this is straightforward: image pixels are continuous values, meaning we can compute the gradient of the loss with respect to the input image and update the pixels directly using gradient descent.
In natural language processing, this process hits a wall. Text is fundamentally discrete. You cannot add a fractional increment to a word token, nor can you compute a meaningful derivative directly with respect to a discrete word choice. Traditional text attacks rely on heuristic word swaps, synonym dictionaries, or genetic algorithms. These methods suffer from a combinatorial explosion, requiring thousands of model queries to find a single successful perturbation.
To bypass this bottleneck, researchers shift the optimization problem from the discrete token space to the model's internal continuous embedding space. By treating text as a continuous vector trajectory rather than a sequence of static tokens, we can leverage the full power of gradient-based optimization.
The Continuous Relaxation of Text
When a language model processes text, its first step is to map discrete tokens into high-dimensional continuous vectors via an embedding layer. Instead of searching for discrete token replacements in a vocabulary, we can optimize a continuous perturbation vector $\delta$ within this embedding space.
Discrete Tokens: [ "The" ] [ "movie" ] [ "was" ] [ "good" ]
│ │ │ │
Embedding Layer: ▼ ▼ ▼ ▼
Continuous Space: [ E("The") ] [ E("movie") ] [ E("was") ] [ E("good") + δ ]
▲
Optimize δ via Gradients
This continuous relaxation fundamentally alters the optimization landscape. High-dimensional embedding spaces provide highly connected, smooth paths toward successful perturbations that are completely hidden from discrete word-swapping heuristics. As Leo Schwinn et al. demonstrate in Adversarial Attacks and Defenses in Large Language Models: Old and New Threats, using unconstrained sign gradient descent on model embeddings allows for rapid identification of adversarial inputs, often requiring fewer than ten iterations.
This efficiency gap means that safety evaluations relying solely on discrete text inputs can severely overestimate a model's security. David Dobre et al. show in Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space that continuous perturbations bypass safety guardrails with a 100% success rate across multiple open-source models, executing in seconds rather than the hours required for discrete token searches.
Bridging the Continuous-to-Discrete Gap
Optimizing vectors in a continuous space is highly efficient, but it introduces a new challenge: if an attacker needs to output a human-readable prompt, or if a defender needs to understand the attack, how do we map these optimized continuous vectors back into discrete tokens?
Researchers have developed three primary strategies to bridge this gap:
1. Nearest-Neighbor Projection
The most direct approach is to project the optimized continuous vector back onto the nearest valid token embeddings in the vocabulary. Sahar Sadrizadeh et al. utilize this in Block-Sparse Adversarial Attack to Fool Transformer-Based Text Classifiers by using cosine similarity to map continuous gradient updates back to valid vocabulary tokens. This ensures that the resulting adversarial inputs consist of real words that remain semantically related to the original text. Boxin Wang et al. employ a similar optimization-then-mapping pipeline in SemAttack: Natural Textual Attacks via Different Semantic Spaces to drastically reduce the number of model queries needed to find valid word substitutions.
2. Distribution Matching
Instead of optimizing unconstrained vectors and projecting them afterward, attacks can be regularized during optimization to ensure they stay close to the manifold of real language. In ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings, Hao Wang et al. introduce a distribution-matching loss, such as Maximum Mean Discrepancy (MMD). This loss forces the optimized continuous vectors to remain close to the valid vocabulary distribution, allowing them to be translated into fluent, stealthy natural language prompts.
3. Latent Reflection
Rather than using explicit mathematical projections, some frameworks delegate the translation back to the language model itself. Ran Li proposes this concept in LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs. LARGO optimizes "adversarial thoughts" directly within the continuous latent space and then prompts the language model to translate these abstract internal states into coherent, human-readable text.
Accelerating Adversarial Training and Defense
The speed of continuous optimization is not just a tool for exploitation; it is a prerequisite for scalable defense. Traditional adversarial training—where a model is iteratively trained on adversarial examples to build robustness—is notoriously slow in NLP because generating discrete text attacks at every training step is computationally prohibitive.
By shifting the defense to the continuous domain, adversarial training becomes highly practical. Instead of searching for discrete word replacements, models can be regularized by injecting continuous noise vectors directly into the embedding layers during the forward pass.
Standard Training: Tokens ──► Embeddings ───────────────────────────► Loss ──► Update Weights
│
Continuous Defense: Tokens ──► Embeddings ──► Add Gradient Noise (δ) ──► Loss ──► Update Weights
Early foundational work by Takeru Miyato et al. in Adversarial Training Methods for Semi-Supervised Text Classification demonstrated that calculating worst-case noise vectors in the continuous embedding space enforces local smoothness, allowing models to better capture semantic meaning and resist perturbations.
Crucially, this continuous robustness transfers back to the discrete world. Gauthier Gidel et al. show in Efficient Adversarial Training in LLMs with Continuous Attacks that models trained against continuous perturbations within a bounded epsilon-ball successfully generalize to resist discrete, token-based adversarial attacks. This eliminates the need for expensive combinatorial token searches during training.
This continuous regularization strategy has been adapted across several domains:
- Synonym Substitution Defenses: Yi Zhou et al. develop Defense against Synonym Substitution-based Adversarial Attacks via Dirichlet Neighborhood Ensemble, treating the synonym space as a continuous convex hull for sampling virtual sentences, making the defense fully differentiable.
- Backdoor Removal: Weiyu Sun et al. introduce BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models, which operates in the continuous embedding space to identify a universal vector drift caused by malicious triggers, neutralizing hidden backdoors without destroying general model utility.
- Knowledge Graph Regularization: Pasquale Minervini et al. demonstrate in Adversarial Sets for Regularising Neural Link Predictors that optimizing over the continuous space of entity embeddings allows an adversary to efficiently find and penalize logical rule violations during training.
The Multimodal and Activation Dimension
The insight that continuous interfaces are highly vulnerable to gradient-based search extends beyond input embeddings to other continuous vectors within modern architectures, such as vision-language interfaces and internal hidden activations.
In multimodal models, image pixels represent a massive, unconstrained continuous input space that maps directly into the same latent space as text embeddings. Anas Awadalla et al. highlight this vulnerability in Are aligned neural networks adversarially aligned?, showing that vision-language models are trivial to jailbreak using adversarial images because the pixel space allows for direct backpropagation.
Haoxuan Ji et al. exploit this continuous shortcut in Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak, optimizing perturbations in the continuous pixel space to discover vulnerabilities 30 times faster than coordinate gradient methods on text, and subsequently translating those visual exploits back into effective text prompts.
Finally, this vulnerability scales predictably when moving deeper into the network's internal activations. Stanislav Fort uncovers a strict linear scaling law in Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens: the number of output tokens an attacker can precisely dictate grows linearly with the number of continuous internal activations they control. By manipulating the full dimensionality of a single internal activation vector rather than a discrete token, an attacker gains enough steering leverage to force the generation of dozens of specific subsequent tokens.
Key Foundational Work
Adversarial Examples on Discrete Sequences for Beating Whole-Binary Malware Detection (2018) — contributed 1 concept elements to this direction.
Adversarial Training Methods for Semi-Supervised Text Classification (2017) — contributed 1 concept elements to this direction.
Adversarial Texts with Gradient Methods (2018) — contributed 1 concept elements to this direction.