Cross-Task Defense: Instruction-Tuning LLMs for Content Safety

Yu Fu, Wen Xiao, Jia Chen, Jiachen Li, Evangelos E. Papalexakis, Aichi Chien, Yue Dong

Introduction

As Large Language Models (LLMs) are increasingly integrated into complex workflows, their safety mechanisms are being tested against more sophisticated threats. While significant research has focused on preventing "jailbreaks" via short, adversarial prompts, a more subtle vulnerability exists: malicious content embedded within long-form documents. This paper, "Cross-Task Defense: Instruction-Tuning LLMs for Content Safety," investigates how LLMs often "forget" their safety training when performing standard NLP tasks—such as summarization or translation—on harmful source material. To address this, the authors propose a defense strategy based on instruction tuning that generalizes safety across various tasks.

The Cross-Task Vulnerability

The researchers identify a "cross-task" vulnerability where the model prioritizes task utility over content safety. For instance, if a user provides a detailed manual for illicit activity and asks the model to "summarize this in three sentences," the model may comply with the summarization request, effectively bypassing its safety filters. This presents two primary challenges:

  1. Detection: Recognizing harmful intent when it is buried within a long, seemingly benign context.
  2. The Utility-Safety Trade-off: Implementing a defense that blocks harmful content without causing the model to become "over-refined," which leads to the refusal of legitimate, safe tasks.

Methodology: Defense via Instruction Tuning

The proposed defense framework utilizes Instruction Tuning to teach models how to handle task instructions paired with malicious documents. The researchers fine-tuned Llama-1-7B and Llama-2-7B models using Low-Rank Adaptation (LoRA).

Training and Loss Functions

To optimize the models, the team developed both single-task and mixed-task loss functions. These functions were designed to minimize the likelihood of the model generating a helpful response to a malicious document while maximizing the probability of a standard refusal.

Diverse Refusal Generation

To prevent the model from overfitting to a single refusal template (e.g., "I cannot fulfill this request"), the authors used a strongly aligned Llama-2-7B model to generate a diverse set of refusal responses. This ensures the model learns the underlying safety principle rather than just a specific output string.

Experimental Setup

The study utilized a specialized defense dataset and several evaluation benchmarks:

  • Defense Training Set: 2,000 malicious documents (averaging ~700 tokens) covering various illicit activities.
  • Task-Harmful Test Set: 100 safety-sensitive documents to measure defense effectiveness.
  • Utility Test Sets: 100 benign documents from the BeaverTails dataset and 100 news articles from CNN/Daily Mail to ensure the model remains functional for safe tasks.
  • Benign Augmentation: 20,000 examples from the Alpaca dataset were mixed into training to preserve general model utility.

Key Insights

Summarization as a Strategic Leverage Point

The researchers discovered that summarization is the task most susceptible to misuse. However, this weakness is also a strategic opportunity: training a defense specifically on summarization tasks yielded the best "transfer" of safety to other tasks like translation or sentiment analysis. This suggests that summarization acts as a "weakest link" that, when patched, strengthens the entire model.

Data Efficiency and the "Sweet Spot"

More data did not linearly correlate with better safety. The study found that 500 defense examples were often optimal. Increasing the count to 2,000 sometimes led to diminishing returns or caused the model to become overly defensive, refusing benign tasks.

Model Resilience

Architecture plays a significant role in safety. Llama-2 proved much more resilient than Llama-1. While Llama-1 often struggled to balance safety and utility—frequently refusing benign tasks after safety tuning—Llama-2 maintained a high pass rate for useful tasks while successfully blocking harmful ones.

Results

  • Safety Improvement: For Llama-2-7B, instruction tuning with 500 examples reduced the "pass rate" (successful processing of malicious content) for summarization from 93.5% to 38.0%.
  • Cross-Task Generalization: Models trained to defend during summarization showed improved safety in Case Conversion and Next Sentence Prediction, despite not being explicitly trained on those tasks.
  • Utility Preservation: Llama-2 maintained a >90% pass rate on benign documents, whereas Llama-1’s utility dropped significantly as the number of defense examples increased.

Conclusion and Future Work

This research demonstrates that instruction tuning is a viable path toward securing LLMs against cross-task vulnerabilities. While the defense significantly reduces the success rate of malicious prompts, it does not eliminate the risk entirely. Future work will explore the "optimal mixing" of defense examples to maximize data efficiency and investigate whether weighting specific tasks like summarization more heavily can further improve safety generalization across all NLP tasks.

Summary updated

Abstract

Recent studies reveal that Large Language Models (LLMs) face challenges in balancing safety with utility, particularly when processing long texts for NLP tasks like summarization and translation. Despite defenses against malicious short questions, the ability of LLMs to safely handle dangerous long content, such as manuals teaching illicit activities, remains unclear. Our work aims to develop robust defenses for LLMs in processing malicious documents alongside benign NLP task queries. We introduce a defense dataset comprised of safety-related examples and propose single-task and mixed-task losses for instruction tuning. Our empirical results demonstrate that LLMs can significantly enhance their capacity to safely manage dangerous content with appropriate instruction tuning. Additionally, strengthening the defenses of tasks most susceptible to misuse is effective in protecting LLMs against processing harmful information. We also observe that trade-offs between utility and safety exist in defense strategies, where Llama2, utilizing our proposed approach, displays a significantly better balance compared to Llama1.
BibTeX
@misc{fu2024cross,
  author = {Yu Fu and Wen Xiao and Jia Chen and Jiachen Li and Evangelos E. Papalexakis and Aichi Chien and Yue Dong},
  title = {{Cross-Task Defense: Instruction-Tuning LLMs for Content Safety}},
  year = {2024},
  eprint = {2405.15202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.15202},
}
Figure