Detecting Adversarial Prompts via Linguistic Fluency

38 concepts34 papers1 proposal

Most adversarial attacks on Large Language Models (LLMs) share a common flaw: they do not look like natural human language. Whether it is a "jailbreak" suffix optimized to bypass safety filters or a "backdoor" trigger designed to misclassify text, these malicious inputs often manifest as ungrammatical, high-entropy, or nonsensical token strings.

Researchers have converged on a powerful intuition to defend these systems: if a sequence of text is statistically improbable according to a standard language model, it is likely an adversarial anomaly. This approach, often termed perplexity-based filtering, treats linguistic fluency as a security boundary.

The Statistical Signature of Optimization

Adversarial attacks like the Greedy Coordinate Gradient (GCG) or the Greedy Coordinate Query (GCQ) work by searching for a specific sequence of tokens that maximizes the probability of a target model producing a harmful response. Because this search is performed over a discrete space of millions of possible tokens, the resulting "suffixes" are often gibberish—strings like "informative supercomplete impeccable ovated" that lack semantic coherence.

As Schwarzschild et al. observe, these algorithmically crafted prompts exhibit a distinct statistical signature. By measuring the perplexity of an input—the degree of "surprise" a reference language model feels when seeing the text—defenders can identify these attacks. A natural human query typically has low perplexity because it follows established patterns of grammar and usage. In contrast, Borevkovic et al. and Zizzo et al. show that adversarial suffixes deviate so significantly from these patterns that they can be flagged and blocked before they ever reach the target model.

Identifying Outliers through Token Removal

While global perplexity checks the entire prompt, more granular methods can pinpoint specific malicious insertions. This is particularly effective against backdoor attacks, where an attacker poisons a training set so that a model misbehaves only when a specific "trigger" word is present.

Qi et al. proposed a framework called ONION (Outlier detectioN for poisoned iNput) based on the idea that a trigger word is an "outlier" that breaks a sentence's fluency. The mechanism is simple:

  1. Calculate the perplexity of the original sentence.
  2. Remove one word at a time and recalculate the perplexity.
  3. If removing a specific word causes a massive drop in perplexity (meaning the sentence became much more "natural" without it), that word is flagged as a trigger.

Yang et al. confirm that rare word triggers create measurable spikes in uncertainty, making them easy for automated filters to strip away at inference time.

Beyond Perplexity: Masked Probabilities

In complex environments like Retrieval-Augmented Generation (RAG), attackers may try to inject "poisoned" documents into a corpus to manipulate the LLM's retrieved context. Even when these documents are designed to appear fluent, the specific tokens optimized to "cheat" the retriever often have lower statistical likelihoods than their neighbors.

Kim et al. introduced GMTP (Gradient-based Masked Token Probability) to address this. Instead of looking at the sequence as a whole, they use a Masked Language Model to predict the probability of each token given its context. Poisoned tokens exhibit significantly lower masked-token probabilities, creating a clear statistical separation that allows the system to filter out over 90% of malicious documents.

The Evasion Arms Race

The primary weakness of perplexity-based filtering is the trade-off between security and usability. As Dong et al. note, legitimate human queries are highly diverse; setting a detection threshold strict enough to catch all gibberish may lead to high false-positive rates for creative or technical user prompts.

Furthermore, attackers are actively developing "stealthy" methods to bypass these filters by prioritizing fluency:

  • Synonym Swapping: Mura and Pintor (LatentBreak) navigate the model's latent space to find synonyms that trigger the same adversarial behavior without breaking the sentence's natural flow.
  • Translation Frameworks: Wang et al. (ASETF) optimize attacks in a continuous embedding space and then "translate" them into fluent natural language.
  • Adversarial Prompters: Di et al. (ProAdvPrompter) train separate LLMs specifically to generate jailbreaking suffixes that are human-readable and low-perplexity.

Despite these advancements in evasion, statistical filtering remains a foundational baseline defense. By forcing attackers to maintain linguistic naturalness, these filters significantly constrain the "attack surface," making it much harder for adversaries to find the precise, high-impact sequences needed to compromise a model.

Key Foundational Work

Rethinking Stealthiness of Backdoor Attack against NLP Models (2021) — contributed 3 concept elements to this direction.
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks (2021) — contributed 2 concept elements to this direction.
Rethinking Textual Adversarial Defense for Pre-trained Language Models (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perplexity_based_filtering_a8a26054,
  title = {perplexity-based filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-based-filtering/rdr_a8a26054667c4a1f9024d53fd1117f1d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_perplexity_based_filtering_a8a26054_v1,
  title = {perplexity-based filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-based-filtering/rdr_a8a26054667c4a1f9024d53fd1117f1d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake