Investigating Coverage Criteria in Large Language Models: An In-Depth Study Through Jailbreak Attacks

Shide Zhou, Tianlin Li, Kailong Wang, Yihao Huang, Ling Shi, Yang Liu, Haoyu Wang

Introduction

The rapid proliferation of large language models (LLMs) has revolutionized artificial intelligence, yet their deployment in sensitive sectors like healthcare and finance is hindered by significant security concerns. One of the most pressing issues is the "jailbreak attack"—carefully crafted prompts designed to bypass safety filters and elicit harmful content. While traditional software engineering relies on coverage criteria (e.g., line or branch coverage) to assess test adequacy, it has been unclear whether these metrics can effectively scale to the billions of parameters in modern LLMs. This study investigates the efficacy of neuron-level coverage criteria in identifying vulnerabilities and detecting real-time jailbreak attempts.

Methodology

The researchers conducted a multi-dimensional empirical study across four diverse LLMs: OPT-125M, Llama-2-7B-Chat, Pythia-12B, and Gemma-2-27B-it. The evaluation framework assessed coverage criteria across three levels: criterion, layer, and token.

Overview of the multidimensional evaluation framework for LLM coverage criteria.

The study utilized benchmarks including Alpaca-GPT4 for normal queries, AdvBench for rejected behaviors, and JailBreakV-28k for successful attacks. A custom dataset of synonymous queries was also generated to test the metrics' ability to identify redundant test cases. The criteria evaluated included Neuron Coverage (NC), Top-K Neuron Coverage (TKNC), TensorFuzz Coverage (TFC), and Neural Coverage (NLC).

Key Insights: The "Logic" of Attacks

A central finding of this research is that LLMs process normal and malicious queries through distinct internal "logic" paths. By performing k-means clustering on the hidden states of the models, the researchers observed that as information progresses through the transformer blocks, the representations of normal, rejected, and jailbreak queries become increasingly distinct.

Clustering analysis of hidden states across different transformer blocks, showing the separation of query types.

Further analysis revealed two critical architectural insights:

  1. Attention Layer Sensitivity: Attention layers were found to be significantly more sensitive to input diversity and malicious intent than MLP layers. This suggests that the global dependencies captured by attention mechanisms are better indicators of a query's underlying "intent."
  2. The Significance of the Last Token: Coverage analysis is most effective when measured at the final token of the input query. The researchers found that "Relative Coverage Growth" actually decreases during the generation of response tokens, as the output becomes more diverse and less tied to the specific structure of the initial attack.

Results and Applications

The insights gained from coverage criteria analysis led to several practical applications for LLM security:

  • Real-time Jailbreak Detection: By training a simple MLP classifier on neural activation features, the authors developed a detector that achieved an average accuracy of 93.61%. This "white-box" approach significantly outperforms traditional "black-box" methods like perplexity filters.
  • Test Case Prioritization: The metrics effectively identified redundant (synonymous) test cases, allowing for more efficient security testing. The method achieved between 84.6% and 90.6% accuracy in distinguishing unique tests from redundant ones.
  • Coverage-Guided Attack Generation: Using coverage as a fitness function for evolving prompts, the researchers were able to generate nearly triple the number of successful jailbreaks compared to random mutation strategies (32 vs. 12).

Conclusion

This study demonstrates that traditional coverage criteria, when adapted for the scale of LLMs, provide a powerful lens for understanding and securing these models. By leveraging internal neuron activation patterns, developers can build more robust real-time detection systems and more efficient security testing suites. While the "white-box" nature of these methods presents challenges for closed-source APIs, they offer a critical foundation for the development of resilient, transparent, and secure AI systems.

Summary updated

Abstract

The swift advancement of large language models (LLMs) has profoundly shaped the landscape of artificial intelligence; however, their deployment in sensitive domains raises grave concerns, particularly due to their susceptibility to malicious exploitation. This situation underscores the insufficiencies in pre-deployment testing, highlighting the urgent need for more rigorous and comprehensive evaluation methods. This study presents a comprehensive empirical analysis assessing the efficacy of conventional coverage criteria in identifying these vulnerabilities, with a particular emphasis on the pressing issue of jailbreak attacks. Our investigation begins with a clustering analysis of the hidden states in LLMs, demonstrating that intrinsic characteristics of these states can distinctly differentiate between various types of queries. Subsequently, we assess the performance of these criteria across three critical dimensions: criterion level, layer level, and token level. Our findings uncover significant disparities in neuron activation patterns between the processing of normal and jailbreak queries, thereby corroborating the clustering results. Leveraging these findings, we propose an innovative approach for the real-time detection of jailbreak attacks by utilizing neural activation features. Our classifier demonstrates remarkable accuracy, averaging 96.33% in identifying jailbreak queries, including those that could lead to adversarial attacks. The importance of our research lies in its comprehensive approach to addressing the intricate challenges of LLM security. By enabling instantaneous detection from the model's first token output, our method holds promise for future systems integrating LLMs, offering robust real-time detection capabilities. This study advances our understanding of LLM security testing, and lays a critical foundation for the development of more resilient AI systems.
BibTeX
@inproceedings{zhou2025investigating,
  author = {Shide Zhou and Tianlin Li and Kailong Wang and Yihao Huang and Ling Shi and Yang Liu and Haoyu Wang},
  title = {{Investigating Coverage Criteria in Large Language Models: An In-Depth Study Through Jailbreak Attacks}},
  booktitle = {ICSE},
  year = {2025},
  doi = {10.1109/icse55347.2025.00209},
}
Figure