Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro

Abstract

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.

Introduction

Large language models (LMs) are typically trained on massive, unfiltered internet datasets, leading them to frequently generate toxic, biased, or offensive text. While retraining from scratch on filtered data is prohibitively expensive and decoding-time interventions (like word filtering) significantly slow down inference, domain-adaptive training—fine-tuning a pre-trained model on a curated, non-toxic dataset—offers a promising middle ground. This paper investigates the limits of this approach across model scales up to 530 billion parameters, exploring efficient corpus generation and parameter-efficient fine-tuning strategies.

Methodology: Self-Generation Enabled domain-Adaptive Training (SGEAT)

A core challenge in detoxification is obtaining a high-quality, non-toxic training corpus that matches the model's distribution. The authors propose Self-Generation Enabled domain-Adaptive Training (SGEAT). Instead of relying on external datasets, SGEAT leverages the LM's own generative capabilities to create a fine-tuning corpus.

The SGEAT pipeline: from prompt construction to fine-tuning on filtered self-generated data.

As illustrated in Figure 1, the process involves three main steps:

  1. Prompt Construction: Using various prompts (including "augmented" prompts derived from the model's own non-toxic unconditional generations) to trigger diverse outputs.
  2. Self-generation and Filtering: The LM generates a large volume of text, which is then passed through an automated classifier (Perspective API). Only non-toxic generations are retained.
  3. Fine-tuning: The LM is fine-tuned on this self-generated, filtered corpus.

This approach mitigates "exposure bias" because the training data perfectly captures the high-density regions of the specific LM's output space, making it more data-efficient than external curated corpora.

Scaling and Parameter Efficiency

The study evaluates detoxification across a family of GPT-3-like models (Megatron-Turing NLG) ranging from 126M to 530B parameters. Two key scaling insights emerged:

  • Baseline Toxicity is Scale-Invariant: Models of all sizes exhibit similar toxicity levels when pre-trained on the same corpus.
  • Detoxification Difficulty Scales: Larger models are more "stubborn" and require larger training corpora to effectively unlearn toxic patterns absorbed during pre-training.

To handle the computational cost of fine-tuning 530B parameters, the authors compared whole-model adaptation with parameter-efficient methods like Adapters (trainable bottleneck layers) and Prefix-tuning. Adapters were found to be superior, successfully steering the model away from toxicity while preserving core language modeling quality and downstream task accuracy.

Results

SGEAT consistently outperformed standard Domain-Adaptive Pre-training (DAPT) across all scales. For the 126M model, SGEAT (augmented) reduced toxicity probability from 57% to 30%. For the 530B model, it reduced the probability from 59% to 43%.

Trade-off between expected maximum toxicity and perplexity (PPL) for SGEAT and DAPT baselines.

As shown in Figure 2, SGEAT achieves a significantly better trade-off between toxicity reduction and model perplexity (PPL) compared to DAPT, even when using only one-third of the training data (50k vs 150k samples). Furthermore, using Adapters with the 530B model during SGEAT actually improved average downstream task accuracy by 0.7% compared to full-model fine-tuning, demonstrating that parameter efficiency can prevent "catastrophic forgetting."

Limitations and Conclusion

While effective, SGEAT is limited by the biases of the automated filters used (e.g., Perspective API), which may disproportionately flag non-toxic text from marginalized groups. Future work should focus on developing fairer toxicity detectors and inherently bias-free foundation models. In summary, this work demonstrates that self-generation and parameter-efficient adapters provide a scalable, efficient path toward detoxifying the world's largest language models.

Summary updated

Abstract

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.
BibTeX
@inproceedings{wang2022exploring,
  author = {Boxin Wang and Wei Ping and Chaowei Xiao and Peng Xu and Mostofa Patwary and Mohammad Shoeybi and Bo Li and Anima Anandkumar and Bryan Catanzaro},
  title = {{Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=mKOSIU5CZy},
}
Figure