Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents

San Kim, Gary Geunbae LeeNAACL 2024 · DOI

TL;DR

This paper introduces Adversarial DPO (ADPO), a novel training method that leverages harmful data to make dialogue agents more resilient to toxic outputs, while maintaining coherence and minimizing evasiveness. ADPO improves upon existing Direct Preference Optimization (DPO) by incorporating an "inner toxic model" to generate and penalize unsafe responses, leading to more robust and safer AI interactions.

What Problem Does This Paper Address?

Large language models (LLMs) have revolutionized dialogue systems, but a significant challenge remains: their propensity to generate toxic or harmful content. While methods like Reinforcement Learning from Human Feedback (RLHF) aim to align AI with human preferences, they can be computationally expensive, require extensive human annotation, and sometimes fail to effectively reduce toxicity. Direct preference optimization (DPO) offers a more stable alternative, but simply filtering out toxic data can lead to a performance trade-off and doesn't fully address the problem. This paper tackles the challenge of reducing toxicity in dialogue agents without sacrificing their coherence or making them overly evasive, a common pitfall in toxicity mitigation.

Methods

The core of this work is Adversarial DPO (ADPO), an extension of the Direct Preference Optimization (DPO) algorithm. ADPO's key innovation is the use of an "inner toxic model". This model is fine-tuned on a dataset augmented with a special "[TOXIC]" token, enabling it to intentionally generate toxic responses.

Here's how it works:

  1. Supervised Fine-Tuning (SFT): The base LLM is first fine-tuned on both normal and toxic dialogues. For toxic dialogues, the "[TOXIC]" token is appended to prompt the model to generate harmful content.
  2. Preference Data Generation: A powerful LLM (Llama2-chat) is used to label responses generated by the fine-tuned model. These labels categorize responses as "chosen" (preferred/safe) or "rejected" (less preferred/unsafe). Crucially, toxic responses are also generated using the inner toxic model.
  3. ADPO Training: ADPO then trains the dialogue agent using these preference labels. It incorporates an additional loss term that penalizes the generation of toxic responses, directly leveraging the outputs from the inner toxic model. This adversarial approach aims to teach the model to actively avoid generating content similar to what the toxic model produces, while still favoring preferred responses.

The ADPO objective function is designed to maximize rewards for preferred responses and minimize penalties for toxic ones, effectively learning to distinguish and avoid harmful outputs.

Datasets

The authors utilized several datasets for their experiments:

  • Helpful and Harmless Human Preference Dataset (Anthropic): This dataset, containing dialogues between humans and AI assistants with preference labels for responses, was used for supervised fine-tuning and generating preference data. The authors used a subset (10%) of this dataset.
  • Bot Adversarial Dialogue (BAD) (Xu et al., 2021): This dataset consists of conversations where crowd workers attempted to elicit harmful responses from an AI. Responses were labeled as safe or unsafe. ADPO was trained on a subset of this dataset (8k dialogues labeled as unsafe).
  • Blended Skill Talk (BST) (Smith et al., 2020): This dataset contains dialogues where participants aimed to demonstrate knowledge, empathy, or specific personas. It was used for evaluating model performance in general conversational contexts.

Key Insights and Ideas

  • Harmful data as a teacher: Instead of just filtering out toxic data, ADPO actively uses it to teach the model what not to do. This "adversarial" approach seems more effective than simply ignoring problematic content.
  • Balancing toxicity and evasiveness: A key challenge in making AI safer is preventing it from becoming overly evasive ("I don't know," "I can't answer"). ADPO's results suggest it can reduce toxicity significantly without a substantial increase in evasiveness, unlike some other methods.
  • Stability in training: ADPO appears to offer a more stable training procedure compared to traditional DPO, especially when dealing with potentially ambiguous preference data.
  • Contextual understanding: By learning from toxic data, ADPO models seem to develop a better understanding of harmful contexts, allowing them to discern and respond more appropriately to potentially unsafe user prompts.

Key Results

  • Toxicity Reduction: ADPO significantly reduced toxicity, achieving nearly a tenfold decrease compared to the base SFT model on the BAD dataset. Toxic metrics fell below 1% in ADPO, while DPO achieved near-zero toxicity.
  • Coherence and Evasiveness: On the BAD dataset, ADPO maintained high coherence (92.6%) and had a low evasiveness score (33.9%), outperforming DPO in coherence and showing only a slight increase in evasiveness compared to the base model. On the BST dataset, ADPO also outperformed DPO in coherence (98.0%) and evasiveness (2.7%), demonstrating its ability to handle general conversations effectively.
  • Efficiency: The authors note that ADPO's approach, by leveraging existing LLMs for annotation and using a more direct training objective, is more data-efficient than methods requiring extensive human labeling.

Limitations and Future Work

  • LLM Utilization: The reliance on LLMs for annotation means that biases or limitations of the annotating LLM could be transferred to the trained model.
  • Data Amount: While ADPO showed promising results with a subset of the Anthropic dataset, using the full dataset and more diverse toxic data could lead to even better performance.
  • Human Annotation Bias: Human evaluations, while correlated with GPT-4, can still be subject to demographic biases.
  • Future Work: The authors suggest exploring various methodologies for effectively utilizing unsafe data, as it contains rich contextual information. They also highlight the need to address the potential conflict between helpfulness and harmfulness, suggesting models should be trained to appropriately decline requests when necessary. Further research into improving both helpfulness and harmlessness is encouraged.

Summary updated

BibTeX
@inproceedings{kim2024adversarial,
  author = {San Kim and Gary Geunbae Lee},
  title = {{Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.118},
}
Figure