Abstract
Warning: This work contains content that may be offensive or upsetting. Eliminating toxicity from large language models (LLMs) is critical to ensure user safety. However, current methods suffer limitations in the analysis and utilization of toxic samples, failing to fully harness their potential. Through comparative analysis of toxic and safe samples, we identified that (i) toxic samples exhibit diversity and (ii) there lies specificity within this diversity. These findings suggest that leveraging these characteristics of toxic samples could enhance the performance of algorithms in LLMs detoxification. Thus, we propose a novel diverse detoxification framework, DivDetox, which comprises two innovative components: a Multi-Category-Induced Personalized Sample Generation (MPSG) strategy and a Scaled Contrastive Direct Preference Optimization (SC-DPO) approach. The former is designed to elicit a variety of personalized toxic responses from LLMs, while the latter is constructed to precisely and fully utilize these toxic responses. Experiments on benchmark datasets across different model scales and various detoxification tasks confirm the effectiveness of our architecture.
Introduction
Despite their transformative capabilities, Large Language Models (LLMs) frequently generate biased or harmful content due to their training on unfiltered web data. Traditional detoxification methods, such as standard Direct Preference Optimization (DPO), often treat toxicity as a binary state. This approach overlooks two critical factors identified in this research:
- The Diversity Gap: Toxicity encompasses a wide range of harms, from hate speech and violent content to PII leaks and bad medical advice. Uniform prompts fail to capture this breadth.
- The Specificity Gap: Different models exhibit unique "toxic personalities." For instance, one model might be prone to medical misinformation while another is more likely to generate biased political commentary.
To address these gaps, we introduce DivDetox, a framework that leverages the inherent diversity and model-specific nature of toxic outputs to achieve superior detoxification.
The DivDetox Framework
DivDetox operates in two primary stages designed to generate and then learn from a model's specific toxic tendencies.
Multi-Category-Induced Personalized Sample Generation (MPSG)
Instead of generic prompts, MPSG uses 11 specific harm categories (e.g., Sexual Content, Privacy, Specialized Advice) paired with in-context examples. This "red-teaming" strategy induces the model to generate a diverse set of toxic responses rooted in its own internal weights. These samples are then filtered and labeled using Perspective API and Llama Guard 2 to create a high-quality, personalized detoxification dataset.
Scaled Contrastive Direct Preference Optimization (SC-DPO)
Standard DPO compares a single safe response to a single toxic one. SC-DPO enhances this through:
- Contrastive Learning: Optimizing the model by comparing one safe sample against multiple diverse toxic samples simultaneously.
- Toxicity Factors: Using a meta-learner to calculate toxicity at both the response and token levels. This allows the loss function to prioritize the most harmful segments of a response.
To maintain efficiency, the framework targets only the second layer of the MLP blocks—the "knowledge region"—allowing for rapid training (e.g., ~84 seconds for a 1.4B parameter model).
Model-Specific Toxicity Patterns
A key insight of this work is that detoxification is most effective when tailored to a model's specific flaws. Topic analysis reveals that while different models might produce similar "safe" responses, their toxic outputs occupy distinct thematic spaces.
As shown in the figure above, Pythia-1.4B and Llama-3-8B exhibit different word associations within the same harm categories, such as "Specialized Advice" and "Privacy." By generating personalized samples, DivDetox directly addresses these model-specific vulnerabilities.
Experimental Results
We evaluated DivDetox using benchmark datasets including RealToxicityPrompts (RTP) and Anthropic Helpful-Harmless (HH).
- Massive Toxicity Reduction: DivDetox reduced toxic generation probability by 93.4% to 99.4% across models ranging from GPT-2 to Llama-3. On Llama-3-8B, toxicity probability dropped to a mere 0.3%, significantly outperforming standard DPO (12.7%).
- Utility Preservation: Unlike methods that "lobotomize" models, DivDetox maintained performance on benchmarks like MMLU. For Llama-3-8B, the MMLU score remained stable (66.3 to 65.9).
- Generalization: Models detoxified on sentence completion tasks showed significant safety improvements in dialogue-based tasks, indicating a generalized learning of safety boundaries.
Conclusion
DivDetox demonstrates that the key to effective LLM detoxification lies in embracing the diversity and specificity of toxic samples. By inducing models to reveal their own harmful tendencies and then applying fine-grained contrastive optimization, we can create safer models without sacrificing their general utility. Future work will explore extending this diversity-led approach to multimodal and multilingual contexts.
Summary updated