Introduction
As Large Language Models (LLMs) are increasingly integrated into complex workflows, their safety mechanisms are being tested against more sophisticated threats. While significant research has focused on preventing "jailbreaks" via short, adversarial prompts, a more subtle vulnerability exists: malicious content embedded within long-form documents. This paper, "Cross-Task Defense: Instruction-Tuning LLMs for Content Safety," investigates how LLMs often "forget" their safety training when performing standard NLP tasks—such as summarization or translation—on harmful source material. To address this, the authors propose a defense strategy based on instruction tuning that generalizes safety across various tasks.
The Cross-Task Vulnerability
The researchers identify a "cross-task" vulnerability where the model prioritizes task utility over content safety. For instance, if a user provides a detailed manual for illicit activity and asks the model to "summarize this in three sentences," the model may comply with the summarization request, effectively bypassing its safety filters. This presents two primary challenges:
- Detection: Recognizing harmful intent when it is buried within a long, seemingly benign context.
- The Utility-Safety Trade-off: Implementing a defense that blocks harmful content without causing the model to become "over-refined," which leads to the refusal of legitimate, safe tasks.
Methodology: Defense via Instruction Tuning
The proposed defense framework utilizes Instruction Tuning to teach models how to handle task instructions paired with malicious documents. The researchers fine-tuned Llama-1-7B and Llama-2-7B models using Low-Rank Adaptation (LoRA).
Training and Loss Functions
To optimize the models, the team developed both single-task and mixed-task loss functions. These functions were designed to minimize the likelihood of the model generating a helpful response to a malicious document while maximizing the probability of a standard refusal.
Diverse Refusal Generation
To prevent the model from overfitting to a single refusal template (e.g., "I cannot fulfill this request"), the authors used a strongly aligned Llama-2-7B model to generate a diverse set of refusal responses. This ensures the model learns the underlying safety principle rather than just a specific output string.
Experimental Setup
The study utilized a specialized defense dataset and several evaluation benchmarks:
- Defense Training Set: 2,000 malicious documents (averaging ~700 tokens) covering various illicit activities.
- Task-Harmful Test Set: 100 safety-sensitive documents to measure defense effectiveness.
- Utility Test Sets: 100 benign documents from the BeaverTails dataset and 100 news articles from CNN/Daily Mail to ensure the model remains functional for safe tasks.
- Benign Augmentation: 20,000 examples from the Alpaca dataset were mixed into training to preserve general model utility.
Key Insights
Summarization as a Strategic Leverage Point
The researchers discovered that summarization is the task most susceptible to misuse. However, this weakness is also a strategic opportunity: training a defense specifically on summarization tasks yielded the best "transfer" of safety to other tasks like translation or sentiment analysis. This suggests that summarization acts as a "weakest link" that, when patched, strengthens the entire model.
Data Efficiency and the "Sweet Spot"
More data did not linearly correlate with better safety. The study found that 500 defense examples were often optimal. Increasing the count to 2,000 sometimes led to diminishing returns or caused the model to become overly defensive, refusing benign tasks.
Model Resilience
Architecture plays a significant role in safety. Llama-2 proved much more resilient than Llama-1. While Llama-1 often struggled to balance safety and utility—frequently refusing benign tasks after safety tuning—Llama-2 maintained a high pass rate for useful tasks while successfully blocking harmful ones.
Results
- Safety Improvement: For Llama-2-7B, instruction tuning with 500 examples reduced the "pass rate" (successful processing of malicious content) for summarization from 93.5% to 38.0%.
- Cross-Task Generalization: Models trained to defend during summarization showed improved safety in Case Conversion and Next Sentence Prediction, despite not being explicitly trained on those tasks.
- Utility Preservation: Llama-2 maintained a >90% pass rate on benign documents, whereas Llama-1’s utility dropped significantly as the number of defense examples increased.
Conclusion and Future Work
This research demonstrates that instruction tuning is a viable path toward securing LLMs against cross-task vulnerabilities. While the defense significantly reduces the success rate of malicious prompts, it does not eliminate the risk entirely. Future work will explore the "optimal mixing" of defense examples to maximize data efficiency and investigate whether weighting specific tasks like summarization more heavily can further improve safety generalization across all NLP tasks.
Summary updated