F²RL: Factuality and Faithfulness Reinforcement Learning Framework for Claim-Guided Evidence-Supported Counterspeech Generation

Haiyang Wang, Yuchen Pan, Xin Song, Xuechen Zhao, Minghao Hu, Bin ZhouEMNLP 2024 · OpenReview

Abstract: Hate speech (HS) on social media exacerbates misinformation and baseless prejudices. Evidence-supported counterspeech (CS) is crucial for correcting misinformation and reducing prejudices through facts. Existing methods for generating evidence-supported CS often lack clear guidance with a core claim for organizing evidence and do not adequately address factuality and faithfulness hallucinations in CS within anti-hate contexts. In this paper, to mitigate the aforementioned, we propose F2RL, a Factuality and Faithfulness Reinforcement Learning framework for generating claim-guided and evidence-supported CS. Firstly, we generate counter-claims based on hate speech and design a self-evaluation mechanism to select the most appropriate one. Secondly, we propose a coarse-to-fine evidence retrieval method. This method initially generates broad queries to ensure the diversity of evidence, followed by carefully reranking the retrieved evidence to ensure its relevance to the claim. Finally, we design a reinforcement learning method with a triplet-based factuality reward model and a multi-aspect faithfulness reward model. The method rewards the generator to encourage greater factuality, more accurate refutation of hate speech, consistency with the claim, and better utilization of evidence. Extensive experiments on three benchmark datasets demonstrate that the proposed framework achieves excellent performance in CS generation, with strong factuality and faithfulness.

The Challenge of Effective Counterspeech

Hate speech (HS) on social media is often built on a foundation of misinformation and prejudice. "Counterspeech"—directly responding to hate speech to reduce its impact—is a vital tool, but it is notoriously difficult to automate effectively. Current Large Language Models (LLMs) frequently suffer from two specific types of "hallucinations" when attempting to counter hate speech:

  1. Factuality Hallucinations: The AI generates fabricated "facts" (e.g., claiming a group donated an impossible sum of money).
  2. Faithfulness Hallucinations: The AI fails to actually rebut the hate speech or ignores the provided evidence entirely, sometimes even drifting into irrelevant historical context.
Examples of factuality and faithfulness hallucinations in automated counterspeech responses.

Furthermore, many existing methods simply "dump" evidence into a response without a clear argumentative structure. F²RL addresses this by providing a framework that generates a central claim first, retrieves supporting evidence for that specific claim, and then uses Reinforcement Learning (RL) to fine-tune the model to stay "faithful" to both the claim and the facts.

The F²RL Framework

The F²RL framework, presented at EMNLP 2024, introduces a three-stage pipeline designed to anchor counterspeech in verifiable facts and coherent logical arguments.

The complete F²RL framework, illustrating the three stages: Self-Evaluation Claim Generation, Coarse-to-Fine Evidence Retrieval, and Factuality/Faithfulness Reinforcement Learning.

1. Self-Evaluation Claim Generation

Instead of generating a full response immediately, the model first produces several potential "counter-claims." It then employs a "self-voting" mechanism where the LLM evaluates its own candidates to select the most effective and objective claim to serve as the backbone of the response. This ensures the response is anchored by a strong, relevant argument.

2. Coarse-to-Fine Evidence Retrieval

To avoid irrelevant data, the system uses the selected claim to generate broad search queries. It retrieves documents from external knowledge bases (like Wikipedia) and then utilizes a reranker—optimized via contrastive learning (InfoNCE loss)—to select the most relevant snippets that support the specific claim.

3. Factuality and Faithfulness Reinforcement Learning

This is the core innovation of the framework. The model is fine-tuned using Proximal Policy Optimization (PPO) with two specialized reward models:

  • Triplet-based Factuality Reward ($r_{fact}$): This model decomposes the generated response into "knowledge triplets" (subject-relation-object) and verifies if these specific units are supported by the retrieved evidence.
  • Multi-aspect Faithfulness Reward ($r_{faith}$): This rewards the model for three distinct criteria: explicitly opposing the hate speech, maintaining consistency with the initial counter-claim, and effectively utilizing the provided evidence.

Key Insights and Results

Claims as Anchors

A significant finding of this research is that retrieval is more effective when based on a counter-claim rather than the original hate speech. This prevents the model from inadvertently echoing the toxic language of the input and focuses the search on constructive evidence.

The Granularity of Truth

By breaking sentences down into triplets for the reward model, the authors move away from subjective, "vibes-based" evaluation. Verifying discrete triplets is a more robust way for a reward model to ensure factual accuracy than evaluating a whole paragraph at once.

Performance Metrics

F²RL was tested across four different LLM backbones (GPT-3.5, GLM4, Qwen1.5, and Llama3) on three benchmark datasets (CONAN, MTCONAN, and MTKGCONAN):

  • Factuality Boost: The framework achieved a 3% to 6% improvement in factual correctness (FA) compared to standard Retrieval-Augmented Generation (RAG).
  • Better Rebuttals: It significantly outperformed baselines in "Faithfulness to Hate Speech" (FH), demonstrating a superior ability to debunk toxic input.
  • Human Validation: In blind tests, human evaluators consistently rated F²RL higher than "Chain of Thought" (CoT) methods, particularly noting its superior use of evidence.

Limitations and Future Directions

While F²RL represents a major step forward, the authors note that it is intended as a tool for human volunteers rather than a fully autonomous bot, as factual errors can still occur. Future work may involve expanding evidence sources beyond Wikipedia to real-time news and exploring multi-agent reinforcement learning, where different AI agents "debate" the best counter-argument before finalizing a response.

Summary updated

Abstract

Hate speech (HS) on social media exacerbates misinformation and baseless prejudices. Evidence-supported counterspeech (CS) is crucial for correcting misinformation and reducing prejudices through facts. Existing methods for generating evidence-supported CS often lack clear guidance with a core claim for organizing evidence and do not adequately address factuality and faithfulness hallucinations in CS within anti-hate contexts. In this paper, to mitigate the aforementioned, we propose F2RL, a Factuality and Faithfulness Reinforcement Learning framework for generating claim-guided and evidence-supported CS. Firstly, we generate counter-claims based on hate speech and design a self-evaluation mechanism to select the most appropriate one. Secondly, we propose a coarse-to-fine evidence retrieval method. This method initially generates broad queries to ensure the diversity of evidence, followed by carefully reranking the retrieved evidence to ensure its relevance to the claim. Finally, we design a reinforcement learning method with a triplet-based factuality reward model and a multi-aspect faithfulness reward model. The method rewards the generator to encourage greater factuality, more accurate refutation of hate speech, consistency with the claim, and better utilization of evidence. Extensive experiments on three benchmark datasets demonstrate that the proposed framework achieves excellent performance in CS generation, with strong factuality and faithfulness.
BibTeX
@inproceedings{wang2024f,
  author = {Haiyang Wang and Yuchen Pan and Xin Song and Xuechen Zhao and Minghao Hu and Bin Zhou},
  title = {{F²RL: Factuality and Faithfulness Reinforcement Learning Framework for Claim-Guided Evidence-Supported Counterspeech Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=7jKX0M4nnl},
}
Figure