In the early days of adversarial machine learning, "breaking" a text classifier was often a matter of brute force. If you could find a sequence of characters or tokens that flipped a model's prediction, the attack was considered a success. However, these early adversarial examples often looked like gibberish—strings of random characters or nonsensical word salads that were trivial for a human to spot or for a simple perplexity filter to block.
A growing body of research has shifted the focus from mere "attack success" to a more difficult, dual-objective problem: How can we generate text that is adversarial to a machine but indistinguishable from natural language to a human?
The solution emerging across dozens of papers is the use of multi-objective optimization—often implemented via Reinforcement Learning (RL) or energy-based decoding—to balance the tension between adversarial impact and linguistic integrity.
The Fundamental Trade-off
The core challenge in modern textual attacks is that the goals of the attacker are inherently in tension. To maximize the chance of bypassing a safety filter or flipping a sentiment label, an optimizer wants to explore the widest possible space of perturbations. However, to remain "stealthy" or "natural," the optimizer must stay within the narrow manifold of human-readable language.
To navigate this, researchers like Amelia Hardy and her colleagues developed ASTPrompter, which explicitly optimizes for both attack success and prompt likelihood. By penalizing high perplexity during the generation process, the system ensures that the resulting adversarial inputs remain natural-sounding and difficult to detect via statistical filters. Similarly, Xuanjie Fang and Yang Liu introduced SDM-ATTACK, which reframes adversarial generation as a sequential decision-making process. Their reward function doesn't just look at whether the model was fooled; it integrates signals for grammatical fluency and semantic preservation using sentence embeddings.
Designing the Composite Reward
The "engine" of this research direction is the multi-objective reward function. Instead of a single scalar value, the generator receives a composite score that acts as a compass, guiding it toward the intersection of "effective" and "plausible."
Typically, these reward functions combine three distinct signals:
- Adversarial Objective: The primary goal, such as maximizing the loss of a classifier, bypassing a safety filter like DALL-E 2's SneakyPrompt, or stripping a watermark in RLCracker.
- Semantic Similarity: Ensuring the "meaning" hasn't changed. Chongyang Gao and Shagufta Mehnaz demonstrate that treating semantic similarity as a primary optimization objective produces examples that are nearly indistinguishable from the original text to human readers Semantic-Preserving Adversarial Example Attack against BERT.
- Linguistic Fluency: Using a secondary language model to calculate perplexity. This ensures the text follows the natural distribution of language, a technique used in the Rewrite and Rollback (R&R) framework by Lei Xu.
Beyond Classification: RAG and Multimodal Stealth
As AI systems have become more complex, so have the multi-objective attacks against them. In Retrieval-Augmented Generation (RAG), an attacker must satisfy an even more complex set of constraints. Hongru Song and Jiafeng Guo developed ReGENT, which uses a multi-objective reward to balance retrieval ranking (getting the document picked), generation influence (forcing the LLM to use the poisoned info), and linguistic naturalness (avoiding detection).
This logic extends to multimodal settings as well. Hao Li and Xu Zhang created CrossGuard, which generates "joint-modal implicit attacks." Their reward system penalizes text that merely describes an image, forcing the model to create sophisticated, reasoning-dependent contexts that trigger harmful behavior only when the text and image are processed together.
Formalizing Constraints via Energy-Based Models
While RL is the most common tool, some researchers have turned to more formal optimization frameworks. Xingang Guo and Lianhui Qin introduced COLD-Attack, which treats jailbreaking as a controllable text generation problem.
Instead of trial-and-error RL, they use a compositional energy function. Each objective—attack success, fluency, and similarity—is represented as an "energy term." By minimizing the total energy, the system can simultaneously satisfy multiple constraints, allowing for the creation of attacks that are not only effective but can be steered toward specific emotional tones or styles.
Defensive Utility: The Non-Zero-Sum Game
The same mechanisms used to attack models are now being used to harden them. Anselm Paulus and Kamalika Chaudhuri proposed AdvGame, which treats safety alignment as a non-zero-sum game between an Attacker and a Defender.
To prevent the adversarial agent from converging on degenerate strategies like generating gibberish, they use a "swapped reward" structure: the attacker is penalized if the defender's response loses its original intent. This ensures that the adversarial prompts remain realistic and challenging, forcing the defender to become more resilient to the types of sophisticated, multi-objective attacks it will face in the real world.