TL;DR
This paper explores whether traditional white-box "coverage criteria" from smaller neural networks can be adapted to evaluate and secure Large Language Models (LLMs). By analyzing internal neuron activations during jailbreak attacks, the authors demonstrate that internal states effectively distinguish malicious queries from normal ones. Leveraging these insights, they develop highly accurate frameworks for real-time jailbreak detection, test case prioritization, and coverage-guided attack generation.
What Problem Does This Paper Address?
As LLMs are increasingly deployed in critical sectors like finance and healthcare, ensuring their safety against malicious exploits—specifically jailbreak attacks—is paramount. Current safety evaluation heavily relies on black-box testing, which evaluates only the final text output. However, black-box testing often fails to uncover hidden vulnerabilities or explain why a model fails.
In traditional software engineering and deep learning testing, "coverage criteria" (like Neuron Coverage) are used to measure how thoroughly a test suite exercises a system's logic. Because LLMs are vastly larger, deeper, and generative, applying these white-box metrics has remained entirely unexplored. This paper addresses this gap by systematically evaluating traditional coverage criteria on LLMs to see if they can improve LLM security testing and defense.
Methods
The authors design a multi-dimensional empirical study to evaluate five traditional deep neural network (DNN) coverage criteria: Neuron Coverage (NC), Top-K Neuron Coverage (TKNC), Top-K Neuron Patterns (TKNP), TensorFuzz Coverage (TFC), and Neural Coverage (NLC).
The workflow of the study is structured around three evaluation dimensions:
- Criterion Level: Assessing which traditional metrics remain computationally feasible and meaningful for LLMs.
- Layer Level: Comparing neuron activations in Attention layers versus Multi-Layer Perceptron (MLP) layers, as well as tracking changes across different transformer blocks.
- Token Level: Investigating how coverage metrics evolve step-by-step during autoregressive token generation.
Using the insights gained from this multi-dimensional analysis, the authors propose three practical security applications:
- Real-time Jailbreak Detection: Training a simple MLP classifier on internal neuron activation features to flag jailbreak attempts at the very first output token.
- Test Case Prioritization: Filtering out redundant test cases (e.g., paraphrased queries) that don't expand neuron coverage, thereby optimizing compute resources.
- Jailbreak Case Generation: A coverage-guided fuzzing approach that iteratively mutates prompts to maximize neuron coverage, systematically uncovering new vulnerabilities.
Datasets
The study utilizes several open-source datasets to construct distinct test suites (Normal, Synonymous, Rejected, and Attack queries):
- Alpaca-GPT4: Used as the source for normal, everyday instruction-following queries.
- JailBreakV-28k: Used to extract pairs of malicious queries and their corresponding jailbreak attack variants.
- AdvBench: Source for straightforward malicious prompts used to trigger model rejection behaviors.
- TruthfulQA, GCG, DeepInception, and Masterkey: Utilized as validation sets to test the generalization capabilities of their jailbreak detector against diverse attack strategies.
The evaluation was performed across four diverse open-source LLMs: OPT-125M, Llama-2-7B-Chat, Pythia-12B, and Gemma-2-27B-it.
Key Insights and Ideas
- Internal States Reveal Intent: A preliminary clustering analysis (using k-means) on the hidden states of LLMs revealed that normal, rejected, and jailbreak queries naturally separate into distinct clusters as information flows through the model. While normal and synonymous queries cluster together early on, jailbreak attacks and standard rejections gradually diverge in the deeper layers of the network.
- Attention Layers are Security Hotspots: Attention layers are significantly more sensitive to variations in query types than MLP layers. Relative Coverage Growth (RCG) metrics showed that Attention layers capture global dependencies and contextual nuances far better, making them the optimal inspection points for white-box testing.
- The "First Token" is Enough: Surprisingly, analyzing neuron coverage at the last token of the original query (before the model generates its response) is the most effective strategy. Generating additional tokens actually diminishes the differences in coverage profiles between normal and attack queries, meaning security checks can be performed efficiently in real-time without waiting for full text generation.
- NC and TKNC Win on Stability: Among the tested criteria, Neuron Coverage (NC) and Top-K Neuron Coverage (TKNC) demonstrated the best stability and generalization across different model architectures, whereas metrics like TKNP and TensorFuzz proved too sensitive to minor text variations.
Key Results
- High-Accuracy Detection: The proposed real-time jailbreak detector achieved an outstanding average accuracy of 93.61% across different models (95.23% on OPT-125M, 93.92% on Llama-2-7B-Chat, and 91.69% on Pythia-12B), significantly outperforming baseline defenses like PARDEN and Perplexity filters on sophisticated template-based attacks.
- Efficient Test Prioritization: The threshold-based prioritization method achieved an average accuracy of 86.26% in successfully filtering out redundant synonymous queries while retaining high-risk attack queries.
- Effective Fuzzing: In jailbreak case generation, the coverage-guided approach consistently discovered more successful jailbreak prompts over multiple iterations compared to a random mutation strategy (generating 32 successful queries in Round 5 vs. only 12 for the random baseline).
Limitations and Future Work
The authors note that while traditional criteria like NC and TKNC are effective, they were originally designed for smaller networks. The immense scale of LLMs means that storing full hidden states for massive datasets remains incredibly resource-intensive (consuming terabytes of data for just a fraction of training sets). Future work includes designing native, LLM-specific coverage criteria that can strike a better balance between comprehensive internal monitoring and computational feasibility, as well as extending white-box exploration to other safety challenges like hallucination and bias detection.
Summary updated