Summary the Savior: Harmful Keyword and Query-based Summarization for LLM Jailbreak Defense

Shagoto Rahman, Ian HarrisNAACL 2025

Sure, here's a blog-style summary of the paper "Summary the Savior: Harmful Keyword and Query-based Summarization for LLM Jailbreak Defense":

TL;DR

This paper introduces "Summary the Savior," a novel defense mechanism against LLM jailbreaking attacks. It leverages harmful keyword and query-based summarization to create security-aware summaries of prompts, which are then classified to detect and block malicious inputs. This approach offers a robust, attack-independent defense that maintains LLM helpfulness.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful tools, but they are vulnerable to "jailbreak attacks." These attacks exploit LLMs to generate inappropriate, harmful, or unsafe content, bypassing their built-in safety measures. Existing defenses often target specific attack types or are easily fooled by subtle prompt variations and natural language ambiguities. This leaves a critical need for a universal, robust defense that can handle diverse attacks and linguistic nuances without sacrificing the LLM's helpfulness.

Methods

The core of "Summary the Savior" lies in its multi-stage approach:

  1. Harmful Keyword Extractor: This component, fine-tuned on a custom dataset, identifies and extracts keywords from a prompt that are indicative of harmful intent. This helps focus subsequent analysis on the most critical parts of the prompt.
  2. Security Aware Summarizer: This is a novel summarization technique that takes the extracted harmful keywords and a security-focused query to generate a summary. This summary is designed to highlight the prompt's potential security implications, going beyond a standard summary.
  3. Harm Classifier: The generated security-aware summaries are then fed into a classifier (in this case, SVM) to categorize the prompt into one of three classes: "harmful," "harmless," or a newly introduced "ambiguous harmful" class.
  4. Ambiguous Harmful Class: This is a key innovation. It addresses prompts that are not explicitly harmful but could be interpreted as such depending on context. Instead of outright blocking them (which could reduce helpfulness), these prompts are passed to the LLM with an extra caution message, allowing the LLM to remain helpful while being more aware of potential risks.

Datasets

The authors created two novel datasets:

  • Harmful Keywords Dataset: A dataset of 155K instances generated using GPT-4, incorporating data from various existing jailbreak datasets (DAN, GPTFuzzer, AdvBench, JBB, Alpaca, XSTest, Wild Teaming at Scale, OR-Bench). This dataset was used to fine-tune the harmful keyword extractor.
  • Security Aware Summaries Dataset: A dataset of 90K instances used to fine-tune the security-aware summarizer. This dataset was generated using Llama-3.1 70B for labeling.

The JBB dataset was also used for evaluating the defense against various specific attacks.

Key Insights and Ideas

  • Summaries as a Defense Vector: The paper's central insight is that summaries, when crafted with a security focus, can be highly effective for detecting harmful prompts. This is a novel application of summarization techniques.
  • The "Ambiguous Harmful" Class: Introducing this class is crucial. It allows the system to differentiate between truly harmful prompts and those that are sensitive or context-dependent, enabling a more nuanced and less restrictive defense. This strikes a better balance between security and LLM helpfulness.
  • Focus on Security: By explicitly guiding the summarization process with harmful keywords and security queries, the method can extract and highlight critical security-relevant information that might be missed by standard summarization or prompt classification alone.

Key Results

  • High Classification Accuracy: When using security-aware summaries as features, machine learning and deep learning models achieved approximately 95% accuracy in classifying prompts, a significant improvement over using prompts directly (around 80%).
  • Lowest Attack Success Rates (ASR): "Summary the Savior" demonstrated the lowest attack success rates across various LLMs (Vicuna, Llama-2, GPT-4) and multiple jailbreak attacks (PAIR, GCG, JB-Chat, Prompt with RS), often achieving 0% ASR.
  • Effective Handling of Ambiguous Data: In the PAIR attack scenario, the "ambiguous harmful" class handling by "Summary the Savior" resulted in a 29% ASR for Vicuna (compared to 100% with no defense) and 0% for Llama-2 and GPT-4, while maintaining low refusal rates on benign data.
  • Outperforms Existing Defenses: The method outperformed state-of-the-art defenses like SmoothLLM, Perplexity Filter, Erase and Check, and even LLM-based prompt classifiers like Llama Guard in many scenarios.

Limitations and Future Work

  • Human Involvement in Data Generation: The authors acknowledge that human involvement in data generation and labeling could improve the reasonableness of the process and mitigate mispredictions.
  • LLM Limitations: The Llama-3.1 70B model used for data generation and GPT-4 for keyword extraction have their own inherent limitations.
  • Future Directions: The authors plan to incorporate both human expertise and multiple LLMs for more robust data generation and labeling. They also aim to explore integrating this summarization-based defense into vision-language models.

Summary updated

BibTeX
@inproceedings{rahman2025summary,
  author = {Shagoto Rahman and Ian Harris},
  title = {{Summary the Savior: Harmful Keyword and Query-based Summarization for LLM Jailbreak Defense}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure