Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks

Shide Zhou, Tianlin Li, Kailong Wang, Yihao Huang, Ling Shi, Yang Liu, Haoyu Wang

TL;DR

This paper explores whether traditional white-box "coverage criteria" from smaller neural networks can be adapted to evaluate and secure Large Language Models (LLMs). By analyzing internal neuron activations during jailbreak attacks, the authors demonstrate that internal states effectively distinguish malicious queries from normal ones. Leveraging these insights, they develop highly accurate frameworks for real-time jailbreak detection, test case prioritization, and coverage-guided attack generation.

What Problem Does This Paper Address?

As LLMs are increasingly deployed in critical sectors like finance and healthcare, ensuring their safety against malicious exploits—specifically jailbreak attacks—is paramount. Current safety evaluation heavily relies on black-box testing, which evaluates only the final text output. However, black-box testing often fails to uncover hidden vulnerabilities or explain why a model fails.

In traditional software engineering and deep learning testing, "coverage criteria" (like Neuron Coverage) are used to measure how thoroughly a test suite exercises a system's logic. Because LLMs are vastly larger, deeper, and generative, applying these white-box metrics has remained entirely unexplored. This paper addresses this gap by systematically evaluating traditional coverage criteria on LLMs to see if they can improve LLM security testing and defense.

Methods

The authors design a multi-dimensional empirical study to evaluate five traditional deep neural network (DNN) coverage criteria: Neuron Coverage (NC), Top-K Neuron Coverage (TKNC), Top-K Neuron Patterns (TKNP), TensorFuzz Coverage (TFC), and Neural Coverage (NLC).

The workflow of the study is structured around three evaluation dimensions:

  1. Criterion Level: Assessing which traditional metrics remain computationally feasible and meaningful for LLMs.
  2. Layer Level: Comparing neuron activations in Attention layers versus Multi-Layer Perceptron (MLP) layers, as well as tracking changes across different transformer blocks.
  3. Token Level: Investigating how coverage metrics evolve step-by-step during autoregressive token generation.
Fig. 2: The Workflow of Our Study

Using the insights gained from this multi-dimensional analysis, the authors propose three practical security applications:

  • Real-time Jailbreak Detection: Training a simple MLP classifier on internal neuron activation features to flag jailbreak attempts at the very first output token.
  • Test Case Prioritization: Filtering out redundant test cases (e.g., paraphrased queries) that don't expand neuron coverage, thereby optimizing compute resources.
  • Jailbreak Case Generation: A coverage-guided fuzzing approach that iteratively mutates prompts to maximize neuron coverage, systematically uncovering new vulnerabilities.

Datasets

The study utilizes several open-source datasets to construct distinct test suites (Normal, Synonymous, Rejected, and Attack queries):

  • Alpaca-GPT4: Used as the source for normal, everyday instruction-following queries.
  • JailBreakV-28k: Used to extract pairs of malicious queries and their corresponding jailbreak attack variants.
  • AdvBench: Source for straightforward malicious prompts used to trigger model rejection behaviors.
  • TruthfulQA, GCG, DeepInception, and Masterkey: Utilized as validation sets to test the generalization capabilities of their jailbreak detector against diverse attack strategies.

The evaluation was performed across four diverse open-source LLMs: OPT-125M, Llama-2-7B-Chat, Pythia-12B, and Gemma-2-27B-it.

Key Insights and Ideas

  • Internal States Reveal Intent: A preliminary clustering analysis (using k-means) on the hidden states of LLMs revealed that normal, rejected, and jailbreak queries naturally separate into distinct clusters as information flows through the model. While normal and synonymous queries cluster together early on, jailbreak attacks and standard rejections gradually diverge in the deeper layers of the network.
Fig. 1: Clustering Experiment Analysis Results: We Select the Results of Block4, Block9, Block16, and Block31 for Display. In the Figure, We Use Colors to Distinguish Datasets and Shapes to Represent Clustering Categories
  • Attention Layers are Security Hotspots: Attention layers are significantly more sensitive to variations in query types than MLP layers. Relative Coverage Growth (RCG) metrics showed that Attention layers capture global dependencies and contextual nuances far better, making them the optimal inspection points for white-box testing.
  • The "First Token" is Enough: Surprisingly, analyzing neuron coverage at the last token of the original query (before the model generates its response) is the most effective strategy. Generating additional tokens actually diminishes the differences in coverage profiles between normal and attack queries, meaning security checks can be performed efficiently in real-time without waiting for full text generation.
  • NC and TKNC Win on Stability: Among the tested criteria, Neuron Coverage (NC) and Top-K Neuron Coverage (TKNC) demonstrated the best stability and generalization across different model architectures, whereas metrics like TKNP and TensorFuzz proved too sensitive to minor text variations.

Key Results

  • High-Accuracy Detection: The proposed real-time jailbreak detector achieved an outstanding average accuracy of 93.61% across different models (95.23% on OPT-125M, 93.92% on Llama-2-7B-Chat, and 91.69% on Pythia-12B), significantly outperforming baseline defenses like PARDEN and Perplexity filters on sophisticated template-based attacks.
  • Efficient Test Prioritization: The threshold-based prioritization method achieved an average accuracy of 86.26% in successfully filtering out redundant synonymous queries while retaining high-risk attack queries.
  • Effective Fuzzing: In jailbreak case generation, the coverage-guided approach consistently discovered more successful jailbreak prompts over multiple iterations compared to a random mutation strategy (generating 32 successful queries in Round 5 vs. only 12 for the random baseline).

Limitations and Future Work

The authors note that while traditional criteria like NC and TKNC are effective, they were originally designed for smaller networks. The immense scale of LLMs means that storing full hidden states for massive datasets remains incredibly resource-intensive (consuming terabytes of data for just a fraction of training sets). Future work includes designing native, LLM-specific coverage criteria that can strike a better balance between comprehensive internal monitoring and computational feasibility, as well as extending white-box exploration to other safety challenges like hallucination and bias detection.

Summary updated

Abstract

Large language models (LLMs) have revolutionized artificial intelligence, but their increasing deployment across critical domains has raised concerns about their abnormal behaviors when faced with malicious attacks. Such vulnerability alerts the widespread inadequacy of pre-release testing. In this paper, we conduct a comprehensive empirical study to evaluate the effectiveness of traditional coverage criteria in identifying such inadequacies, exemplified by the significant security concern of jailbreak attacks. Our study begins with a clustering analysis of the hidden states of LLMs, revealing that the embedded characteristics effectively distinguish between different query types. We then systematically evaluate the performance of these criteria across three key dimensions: criterion level, layer level, and token level. Our research uncovers significant differences in neuron coverage when LLMs process normal versus jailbreak queries, aligning with our clustering experiments. Leveraging these findings, we propose three practical applications of coverage criteria in the context of LLM security testing. Specifically, we develop a real-time jailbreak detection mechanism that achieves high accuracy (93.61% on average) in classifying queries as normal or jailbreak. Furthermore, we explore the use of coverage levels to prioritize test cases, improving testing efficiency by focusing on high-risk interactions and removing redundant tests. Lastly, we introduce a coverage-guided approach for generating jailbreak attack examples, enabling systematic refinement of prompts to uncover vulnerabilities. This study improves our understanding of LLM security testing, enhances their safety, and provides a foundation for developing more robust AI applications.
BibTeX
@inproceedings{zhou2025understanding,
  author = {Shide Zhou and Tianlin Li and Kailong Wang and Yihao Huang and Ling Shi and Yang Liu and Haoyu Wang},
  title = {{Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks}},
  booktitle = {ICSE},
  year = {2025},
  doi = {10.1109/icse55347.2025.00209},
}
Figure