Efficient Detection of Toxic Prompts in Large Language Models

Yi Liu, Junzhe Yu, Huijia Sun, Ling Shi, Gelei Deng, Yuqi Chen, Yang Liu

Abstract

Large language models (LLMs) like ChatGPT and Gemini have significantly advanced natural language processing, enabling various applications such as chatbots and automated content generation. However, these models can be exploited by malicious individuals who craft toxic prompts to elicit harmful or unethical responses. These individuals often employ jailbreaking techniques to bypass safety mechanisms, highlighting the need for robust toxic prompt detection methods. Existing detection techniques, both blackbox and whitebox, face challenges related to the diversity of toxic prompts, scalability, and computational efficiency. In response, we propose ToxicDetector, a lightweight greybox method designed to efficiently detect toxic prompts in LLMs. ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP) classifier for prompt classification. Our evaluation on various versions of the LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector achieves a high accuracy of 96.39% and a low false positive rate of 2.00%, outperforming state-of-the-art methods. Additionally, ToxicDetector's processing time of 0.0780 seconds per prompt makes it highly suitable for real-time applications. ToxicDetector achieves high accuracy, efficiency, and scalability, making it a practical method for toxic prompt detection in LLMs.

Introduction

Large Language Models (LLMs) are transformative, but they possess a well-documented vulnerability to adversarial exploitation. Malicious users continuously develop "jailbreaking" techniques—sophisticated prompt engineering designed to bypass safety filters and trick models into generating harmful, unethical, or dangerous content.

Current detection tools generally fall into two categories: blackbox APIs, which are scalable but often miss subtle semantic attacks, and whitebox methods, which are accurate but computationally prohibitive for real-time production environments. In the paper "Efficient Detection of Toxic Prompts in Large Language Models," researchers introduce ToxicDetector, a "greybox" framework that combines the deep insight of internal model states with the speed required for high-traffic applications.

The Challenge of Jailbreaking

The primary difficulty in securing LLMs lies in the diversity and evolution of toxicity. While a simple keyword filter might catch a direct request for illegal information, it often fails against jailbroken prompts that use complex personas or fictional scenarios to disguise malicious intent.

Comparison of a direct toxic prompt versus a jailbroken toxic prompt that successfully bypasses standard safety filters.

Existing methods struggle with this balance. Blackbox methods fail to capture the subtle semantic shifts used in jailbreaking, while whitebox methods analyze internal activations at a cost that makes them impractical for real-time user experiences.

Methodology: The ToxicDetector Framework

ToxicDetector addresses these challenges through a three-stage workflow: concept extraction, feature construction, and classification.

The complete ToxicDetector framework, illustrating the training pipeline using concept augmentation and the real-time jailbreak detection process.
  1. Toxic Concept Extraction & Augmentation: Rather than relying on static keyword lists, the system uses an LLM to generate high-level "concept prompts" (e.g., "Requesting information on committing a crime"). An automated algorithm then creates a diverse set of these concepts to ensure broad coverage across various toxic scenarios.
  2. Feature Extraction: When a user submits a prompt, ToxicDetector extracts the embedding of the last token from every layer of the LLM. By calculating the element-wise product of these embeddings against those of known toxic concepts, the system creates a feature vector representing the similarity of the user's intent to malicious concepts at multiple levels of abstraction.
  3. Classification: These feature vectors are processed by a lightweight Multi-Layer Perceptron (MLP). Because the MLP is small and the embeddings are generated during the LLM's standard inference process, the computational overhead is minimal.

Key Insights and Performance

The effectiveness of ToxicDetector stems from several key design choices:

  • Last Token Significance: Leveraging the final token's embedding captures the semantic meaning of the entire sequence without the need to process the entire activation map, significantly reducing latency.
  • Multi-Layer Depth: By analyzing embeddings from every layer, the model can detect toxic intent whether it is recognized early (syntactic level) or late (semantic level) in the model's processing.
  • Concept Augmentation: Using LLMs to "hallucinate" diverse variations of toxic concepts significantly improves the model's ability to generalize to unseen or subtle attacks.

In evaluations, ToxicDetector achieved an accuracy of 96.39% and a remarkably low false positive rate of 2.00%. Most importantly for production use, its processing time is just 0.0780 seconds per prompt, making it approximately 30x faster than many API-based alternatives.

Conclusion

ToxicDetector represents a significant step forward in LLM safety. By utilizing internal model embeddings through an efficient greybox approach, it provides a robust, real-time defense against evolving jailbreak techniques without compromising the performance or scalability of the underlying language model.

Reference: Liu, Y., Yu, J., Sun, H., Shi, L., Deng, G., Chen, Y., & Liu, Y. (2024). Efficient Detection of Toxic Prompts in Large Language Models. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24).

Summary updated

Abstract

Large language models (LLMs) like ChatGPT and Gemini have significantly advanced natural language processing, enabling various applications such as chatbots and automated content generation. However, these models can be exploited by malicious individuals who craft toxic prompts to elicit harmful or unethical responses. These individuals often employ jailbreaking techniques to bypass safety mechanisms, highlighting the need for robust toxic prompt detection methods. Existing detection techniques, both blackbox and whitebox, face challenges related to the diversity of toxic prompts, scalability, and computational efficiency. In response, we propose ToxicDetector, a lightweight greybox method designed to efficiently detect toxic prompts in LLMs. ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP) classifier for prompt classification. Our evaluation on various versions of the LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%, outperforming state-of-the-art methods. Additionally, ToxicDetector's processing time of 0.0780 seconds per prompt makes it highly suitable for real-time applications. ToxicDetector achieves high accuracy, efficiency, and scalability, making it a practical method for toxic prompt detection in LLMs.
BibTeX
@inproceedings{liu2024efficient,
  author = {Yi Liu and Junzhe Yu and Huijia Sun and Ling Shi and Gelei Deng and Yuqi Chen and Yang Liu},
  title = {{Efficient Detection of Toxic Prompts in Large Language Models}},
  booktitle = {ASE},
  year = {2024},
  doi = {10.1145/3691620.3695018},
}
Figure