Improving Neutral Point of View Text Generation through Parameter-Efficient Reinforcement Learning and a Small-Scale High-Quality Dataset

Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon

Abstract

This paper describes the construction of a dataset and the evaluation of training methods to improve generative large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e., to provide significantly more informative, diverse and impartial answers. The dataset, the SHQ-NPOV dataset, comprises 300 high-quality, human-written quadruplets: a query on a sensitive topic, an answer, an NPOV rating, and a set of links to source texts elaborating the various points of view. The first key contribution of this paper is a new methodology to create such datasets through iterative rounds of human peer-critique and annotator training, which we release alongside the dataset. The second key contribution is the identification of a highly effective training regime for parameter-efficient reinforcement learning (PE-RL) to improve NPOV generation. We compare and extensively evaluate PE-RL and multiple baselines—including LoRA finetuning, SFT, and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\% \rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating good answers from the best answers ($60.25\% \rightarrow 85.21\%$ for presence of supportive details, $68.74\% \rightarrow 91.43\%$ for absence of oversimplification). Finally, our evaluation finds no statistical differences between results on training topics and separated evaluation topics, providing strong evidence that our approach to training PE-RL exhibits very effective out-of-topic generalization.

Introduction

As Large Language Models (LLMs) are increasingly used as primary information sources, their ability to handle sensitive or controversial topics with a Neutral Point of View (NPOV) becomes paramount. Often, models struggle to maintain neutrality, either by hallucinating a consensus, exhibiting training bias, or providing oversimplified answers that lack necessary nuance. Traditional solutions involving massive datasets for Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF) are often prohibitively expensive and prone to overfitting on small datasets. This research explores whether a small-scale, high-quality dataset combined with parameter-efficient training can solve these challenges.

The SHQ-NPOV Dataset

A core contribution of this work is the SHQ-NPOV (Small-Scale High-Quality NPOV) dataset. Rather than focusing on volume, the researchers prioritized quality, curating just 300 query-answer pairs. These were developed by expert writers through iterative rounds of peer-critique and revision to ensure they met strict neutrality standards.

Sample query-answer pairs from the dataset demonstrating the NPOV scoring scale.

The dataset includes 1,190 queries covering 89 in-distribution topics and 30 out-of-distribution (OOD) topics. This diversity allows for rigorous testing of the model's ability to generalize the concept of neutrality to entirely new subjects, ranging from political gerrymandering to alternative medicine.

Methodology: Parameter-Efficient Reinforcement Learning

The researchers proposed a pipeline that leverages Parameter-Efficient Reinforcement Learning (PE-RL). Instead of updating the billions of parameters in a 20B model, they utilized Low-Rank Adaptation (LoRA) adapters. This approach involves training two separate adapters: one for a Reward Model (to recognize neutrality) and one for the Policy (to generate it).

The three-stage pipeline: dataset creation, parameter-efficient reinforcement learning (PE-RL), and comprehensive analysis.

The training recipe began with initializing the model with a LoRA SFT checkpoint trained on the highest-quality examples, followed by refinement via RL using the LoRA reward model. To evaluate performance at scale, a 60B parameter "autorater" was developed, showing 95% alignment with human expert judgments.

Results and Generalization

The PE-RL approach demonstrated significant improvements over traditional baselines. Beyond the headline increase in NPOV quality to 99.08%, the model showed a unique ability to capture sophisticated linguistic markers. The presence of supportive details (facts and statistics) rose from 60.25% to 85.21%, while the absence of oversimplification improved from 68.74% to 91.43%.

Crucially, PE-RL avoided the "memorization" trap. While full-parameter models often saw performance degrade on OOD topics, the PE-RL models maintained consistent performance across all test templates. This suggests the model learned the stylistic essence of neutrality rather than just memorizing facts about the training topics.

Conclusion and Future Work

This study proves that data quality is more critical than quantity for teaching LLMs complex stylistic behaviors like neutrality. By using only 300 expert examples and parameter-efficient training, models can achieve high-quality, generalized NPOV generation.

Future research should address remaining challenges, such as factual hallucinations—as NPOV training makes answers sound balanced but not necessarily more accurate—and the cultural subjectivity of neutrality. Integrating these methods with Retrieval-Augmented Generation (RAG) could further ensure that all significant viewpoints are captured from live, authoritative sources.

Summary updated

Abstract

This paper describes the construction of a dataset and the evaluation of training methods to improve generative large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e., to provide significantly more informative, diverse and impartial answers. The dataset, the SHQ-NPOV dataset, comprises 300 high-quality, human-written quadruplets: a query on a sensitive topic, an answer, an NPOV rating, and a set of links to source texts elaborating the various points of view. The first key contribution of this paper is a new methodology to create such datasets through iterative rounds of human peer-critique and annotator training, which we release alongside the dataset. The second key contribution is the identification of a highly effective training regime for parameter-efficient reinforcement learning (PE-RL) to improve NPOV generation. We compare and extensively evaluate PE-RL and multiple baselines-including LoRA finetuning (a strong baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\%\rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating good answers from the best answers ($60.25\%\rightarrow 85.21\%$ for presence of supportive details, $68.74\%\rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Finally, our evaluation finds no statistical differences between results on topics that appear in the training dataset and those on separated evaluation topics, which provides strong evidence that our approach to training PE-RL exhibits very effective out of topic generalization.
BibTeX
@misc{hoffmann2025improving,
  author = {Jessica Hoffmann and Christiane Ahlheim and Zac Yu and Aria Walfrand and Jarvis Jin and Marie Tano and Ahmad Beirami and Erin van Liemt and Nithum Thain and Hakim Sidahmed and Lucas Dixon},
  title = {{Improving Neutral Point of View Text Generation through Parameter-Efficient Reinforcement Learning and a Small-Scale High-Quality Dataset}},
  year = {2025},
  eprint = {2503.03654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03654},
}
Figure