Detoxifying Online Discourse: A Guided Response Generation Approach for Reducing Toxicity in User-Generated Text

Ritwik Bose, Ian Perera, Bonnie DorrACL 2023

TL;DR

This paper tackles the challenge of reducing toxicity in online discussions while preserving the original meaning and community style. They propose a guided response generation approach using fine-tuned Large Language Models (LLMs) to rephrase toxic user-generated text into more constructive language that mimics the target community's tone.

What Problem Does This Paper Address?

Online platforms are rife with toxic, divisive, and inflammatory language that hinders constructive discourse. This is particularly challenging for topics deeply tied to personal beliefs and identities, where emotional weight can escalate conflict. While LLMs offer potential for reframing such content, current methods often produce generic or overly formal text, lacking the authenticity needed for social media. The authors aim to bridge this gap by developing LLMs that can generate responses that are both non-toxic and stylistically appropriate for specific online communities, acting as an "echo-chamber burster" by presenting opposing views constructively.

Methods

The core of their approach involves fine-tuning LLMs, specifically a T5-Large model, using a guided response generation framework. This framework utilizes prompt-completion tasks structured around parent-child post relationships within online discussions.

Key methodological components include:

  • Enrichment Encoding: Numerical data (like toxicity scores) is translated into textual descriptions to be used in prompts. A toxicity threshold ($\alpha=0.5$) was established to categorize posts as high or low toxicity.
  • Template Construction: Several prompt templates were designed, incorporating parent posts, child posts, and their summaries. These templates vary in the amount of contextual information provided to the model, ranging from just the parent post (PC) to parent and child posts with toxicity and summary information (PCTS).
  • Fine-Tuning: The T5-Large model was fine-tuned on these templates for 5 epochs. They experimented with different configurations, including using summaries generated by a fine-tuned flan-t5-xxl model or ChatGPT, and also compared against a baseline ChatGPT detoxification prompt.

Datasets

The primary dataset used was collected from Reddit communities, specifically r/prochoice and r/prolife, over a year. This collection aimed to capture contrasting and opposing viewpoints.

  • Data Collection: A combination of the PushShift API and the Python Reddit API Wrapper (PRAW) was used. Data included posts and comments, with efforts made to capture deleted content and moderator activity.
  • Enrichment: The collected data was enriched with off-the-shelf tools for classifying emotions, sentiment, and toxicity. Summaries were generated using a fine-tuned flan-t5-xxl model.
  • Dataset Split: A randomly selected training (70%) and test (20%) split was constructed, with careful attention to avoid cross-contamination between parent and child posts in different sets. The final experimental datasets contained a specific number of low and high toxicity posts for both training and testing.

Key Insights and Ideas

  • Authenticity vs. Constructiveness: The authors found that while LLMs can be guided towards more constructive language, achieving authenticity (mimicking community style) is a separate challenge. Simply using ChatGPT for summarization alongside toxicity information didn't necessarily yield the best results.
  • Fine-tuning for Community Style: Fine-tuning LLMs on specific community data is crucial for generating responses that feel natural and appropriate within that community's context, rather than generic or overly formal.
  • Toxicity Measures as Signals: Existing toxicity rating libraries can be helpful signals for reducing toxicity, but they have limitations, such as conflating profanity with toxicity and being slow to adapt to new language trends.
  • The "Echo-Chamber Burster" Concept: The idea of generating posts that present opposing views in a constructive, community-appropriate style is a novel way to potentially foster dialogue across polarized groups.

Key Results

  • Automatic Evaluation (BLEURT): The PCTS models (incorporating parent post, toxicity, and summary) performed comparably to a ChatGPT-only detoxification baseline. Interestingly, using ChatGPT to generate summaries for PCTS did not significantly improve results over using flan-t5-xxl summaries. Reducing toxicity did not significantly impact BLEURT scores, suggesting the metric is less sensitive to constructiveness.
  • Human Evaluation:
    • PCS (Parent-Child with Summaries) achieved the highest scores for fluency (4.06) and authenticity/humanness (3.79).
  • PCTS+ChatGPT (Parent-Child with Toxicity and Summaries, using ChatGPT for rephrasing) achieved the highest scores for toxicity/constructiveness (3.32) and relevance (3.71).
  • This highlights a trade-off: PCS excels at sounding human, while PCTS+ChatGPT is better at making the text constructive.

Limitations and Future Work

  • Toxicity Measure Limitations: The automated toxicity enrichments sometimes conflate profanity with toxicity. The emotionally charged nature of the dataset can lead to subjective toxicity measures. There's a risk of inadvertently increasing toxicity under certain conditions, which requires further study.
  • Resource and Language Availability: Training T5-large models requires significant GPU resources. The 512-token limit can lead to poor rephrasing of longer posts. Performance in non-English languages is dependent on the availability of enrichment and summarization models.
  • Evaluation Drawbacks: The Likert scale used for human evaluation has limitations like inter-annotator agreement variability and a tendency for annotators to choose middle ratings.
  • Future Work:
    • Addressing the limitations of toxicity measures by incorporating "stance predicates" to improve faithfulness to the original text's meaning.
  • Exploring more robust evaluation methods, such as system-level probabilistic assessment (SPA).
  • Investigating the potential for misuse, such as generating more toxic text or creating falsified perspectives.
  • Improving the ability to tune LLMs for specific communities to learn and address toxic patterns.

Summary updated

BibTeX
@inproceedings{bose2023detoxifying,
  author = {Ritwik Bose and Ian Perera and Bonnie Dorr},
  title = {{Detoxifying Online Discourse: A Guided Response Generation Approach for Reducing Toxicity in User-Generated Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure