TL;DR
LLMEffiChecker reveals a critical yet overlooked vulnerability in Large Language Models (LLMs): "efficiency attacks." By introducing tiny, imperceptible perturbations to an input—sometimes just a single character—attackers can delay the generation of the "End of Sentence" (EOS) token, forcing models to hit their maximum iteration limits and increasing latency and energy consumption by over 3,000%.
What Problem Does This Paper Address?
While the AI community has spent years hardening LLMs against accuracy attacks (inputs designed to make the model say something wrong), we have largely ignored efficiency robustness. LLMs are computationally expensive; they generate text iteratively until they either produce a stop token (EOS) or hit a pre-configured maximum length.
The authors identify a "pessimistic threshold" problem: to ensure they don't cut off long, valid responses, most developers set max_length limits (often 300-512 tokens) that are far beyond the average output length. This paper explores how an adversary can exploit this gap to "hide the stop sign," causing the model to spin its wheels unnecessarily, leading to massive resource waste or even Denial-of-Service (DoS) on commercial APIs.
Methods
The authors introduce LLMEffiChecker, a framework designed to find these efficiency-draining inputs in two scenarios:
- White-Box Setting: If the attacker has access to the model's internal weights, LLMEffiChecker uses a gradient-guided technique. It calculates which tokens in the input have the highest "importance score" regarding the probability of the EOS token appearing. It then applies minimal perturbations (character-level typos, token-level swaps, or structural rephrasing) to minimize that probability.
- Black-Box Setting: For proprietary models (like GPT-3.5), the tool uses causal inference and delta debugging. It systematically removes tokens from the input to observe which ones cause the largest jump in output length, identifying "critical tokens" to perturb without needing internal model access.
To defend against these attacks, the authors propose a lightweight SVM-based detector that analyzes the model's internal hidden states (latent representations) during the first few steps of generation to flag and kill "abnormal" inputs early.
Datasets
The study evaluates nine popular LLMs (including Google T5, Facebook FairSeq, and Salesforce CodeGen) across several benchmarks:
- MultiUN: A collection of translated UN documents for multilingual testing.
- ZH19: English-Chinese translation pairs.
- HellaSwag: A dataset for sentence completion.
- MBPP (Mostly Basic Python Problems): Used to test code generation models.
- WMT19: News translation data.
Key Insights and Ideas
- Efficiency is Output-Driven: Unlike traditional software where input size usually dictates processing time, LLM efficiency is almost entirely dependent on the output length, which is non-deterministic and easily manipulated.
- The EOS Trap: The "aha moment" is that you don't need to change the meaning of the input to break the model's efficiency. You just need to break the model's confidence that the sentence is finished.
- One Character is Enough: The most surprising finding is the sensitivity. Inserting a single "b" into a German word increased a translation model's latency from 0.8s to over 20s.
- Latent Space Signatures: Maliciously perturbed inputs look different to the model's internal layers than natural inputs do, even if they look similar to human eyes. This allows for very high-accuracy detection with negligible overhead.
Key Results
- Massive Degradation: Across nine models, LLMEffiChecker increased response latency by 325% to 3,244% and energy consumption by 344% to 3,616%.
- Mobile Impact: In a real-world case study on a Samsung Galaxy S9+, a single-character perturbation caused the model to drain 30 times more battery power than a normal input.
- Commercial Vulnerability: Even "black-box" testing on OpenAI’s GPT-3.5 showed that the tool could increase latency by up to 156% with minimal changes.
- Effective Defense: Their proposed SVM detector achieved over 83% accuracy (and often 100% on specific models) while adding less than 1% to the total processing time.
Limitations and Future Work
The authors acknowledge that their perturbations, while "unnoticeable" to many, may occasionally result in grammatically incorrect tokens (e.g., a typo that creates an out-of-vocabulary word). While this mimics real-world user behavior, future work could focus on ensuring all generated test cases are perfectly fluent.
Additionally, while the study covers nine models, the rapid evolution of LLM architectures (like MoE or long-context windows) means the "pessimistic threshold" might shift, requiring continuous updates to efficiency testing frameworks.
Summary updated