Abstract
It has been widely observed that language models (LMs) respond in predictable ways to algorithmically generated prompts that are seemingly unintelligible. This is both a sign that we lack a full understanding of how LMs work, and a practical challenge, because opaqueness can be exploited for harmful uses of LMs, such as jailbreaking. We present the first thorough analysis of opaque machine-generated prompts, or autoprompts, pertaining to 6 LMs of different sizes and families. We find that machine-generated prompts are characterized by a last token that is often intelligible and strongly affects the generation. A small but consistent proportion of the previous tokens are prunable, probably appearing in the prompt as a by-product of the fact that the optimization process fixes the number of tokens. The remaining tokens fall into two categories: filler tokens, which can be replaced with semantically unrelated substitutes, and keywords, that tend to have at least a loose semantic relation with the generation, although they do not engage in well-formed syntactic relations with it. Additionally, human experts can reliably identify the most influential tokens in an autoprompt a posteriori, suggesting these prompts are not entirely opaque. Finally, some of the ablations we applied to autoprompts yield similar effects in natural language inputs, suggesting that autoprompts emerge naturally from the way LMs process linguistic inputs in general.
Introduction
Language models respond with surprising predictability to algorithmically generated prompts that are completely unreadable to humans. These "autoprompts" are more than just a scientific curiosity; they represent a significant security challenge. If we don't understand how a string of seemingly random characters can trigger a specific model response, we can't effectively defend against "jailbreaking" or adversarial attacks.
While previous research has focused on how to create these prompts (optimization) or how well they transfer between models, this paper asks a more fundamental qualitative question: What is actually inside an autoprompt? The authors aim to demystify these strings by breaking them down into functional components, comparing them to the "natural" prompts that produce the same outputs—the "evil twin" setup.
Methodology
The researchers employed a "minimal pair" approach, finding autoprompts that lead to the same continuation as a natural language sequence. They used the Greedy Coordinate Gradient (GCG) algorithm to optimize these prompts across six models from the Pythia and OLMo families, ranging from 1B to 12B parameters. To deconstruct them, they used several ablation techniques:
- Pruning: A greedy approach where tokens are removed one by one. If the model output remains the same, the token is considered "prunable" (useless noise).
- Substitution: Replacing each token with the 10,000 most frequent words from a large corpus (The Pile or Dolma) to identify "fillers" (easily replaced) versus "keywords" (essential and semantically specific).
- Shuffling: Randomizing the order of tokens to test if the "gibberish" has a hidden syntax.
- Human Annotation: Experts manually labeled tokens as "important" or "not important" to see if human intuition could penetrate the perceived opacity of the prompts.
Functional Components of Autoprompts
The analysis reveals that autoprompts are not monolithic blocks of gibberish but consist of distinct functional parts.
- The "Last Token" Hegemony: In autoregressive models, the very last token of a prompt is the most influential. In autoprompts, this token is often intelligible and has a strong semantic link to the start of the generated text, acting as a bridge.
- Optimization Artifacts: Roughly 20% of tokens in an autoprompt are "prunable." These appear to be by-products of the optimization process, which usually forces a fixed prompt length (e.g., 10 tokens), leading the algorithm to "fill the slots" with noise.
- Fillers vs. Keywords: Non-prunable tokens fall into two camps. Fillers can be replaced by hundreds of unrelated words without changing the output. Keywords have very small "equivalent sets" (few substitutes) and often share a loose semantic meaning with the output.
Key Results
- Prunability and Position: 73.2% of autoprompts could be shortened. On average, a 10-token prompt contained 2.6 useless tokens. Positional analysis shows that tokens closer to the end are significantly less likely to be pruned, reinforcing the importance of the final tokens.
- Order Matters: Shuffling the tokens (even while keeping the last one fixed) caused a massive drop in BLEU scores (from ~1.0 to 0.06). This proves that autoprompts are not just "bags of keywords"; their local ordering is functional.
- Human Intuition: Human experts were surprisingly good at identifying influential tokens a posteriori. Tokens labeled as "important" by humans had significantly smaller equivalent sets (median of 2 substitutes) compared to those labeled "replaceable" (median of 15).
- Natural Link: Bigram analysis showed that while internal autoprompt transitions are rare in natural text, the transition from the last token of the autoprompt to the first token of the generation looks exactly like natural language.
Conclusion
Autoprompts are not as mysterious as they seem. They rely on predictable mechanisms—such as a highly influential last token and semantic "keywords"—suggesting that Large Language Models (LLMs) process these opaque strings using the same underlying logic they use for human language. Future work could explore whether these findings hold for much larger models or for prompts generated through different methods, such as reinforcement learning.
Summary updated