Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Jessica D. Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon

Abstract

The paper shows that parameter-efficient reinforcement learning (PE-RL) is a highly effective training regime to improve large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e. to provide significantly more informative, diverse and impartial answers. This is shown by evaluating PE-RL and multiple strong baselines—including LoRA finetuning (strongest baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\% \rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating sufficient answers from "great" answers ($60.25\% \rightarrow 85.21\%$ for presence of supportive details, $68.74\% \rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Moreover, our evaluation also finds a key property of PE-RL for this task: unlike methods that update all parameters, it generalizes out of topic. Finally, to enable further studies we also release the dataset, SHQ-NPOV, and provide a methodology to create such datasets through iterative rounds of human peer-critique and annotator training.

Introduction

As Large Language Models (LLMs) increasingly serve as primary information interfaces, their tendency to adopt specific biases or oversimplify complex, sensitive issues poses a significant risk of exacerbating social polarization. Teaching models to maintain a Neutral Point of View (NPOV) is challenging due to the scarcity of high-quality training data and the tendency of standard fine-tuning methods to overfit to specific training topics, failing to generalize to new, "out-of-distribution" (OOD) subjects.

Methodology: Parameter-Efficient Reinforcement Learning

To address these challenges, researchers from Google DeepMind propose a pipeline centered on Parameter-Efficient Reinforcement Learning (PE-RL) using LoRA (Low-Rank Adaptation).

The three-stage pipeline: Dataset creation, Parameter-efficient RL, and Analysis.

Instead of updating billions of parameters, the method trains two small LoRA adapters: one for a Reward Model (to score neutrality) and one for the Policy (the LLM generating text). The best results are achieved by initializing the RL policy with a "warm" checkpoint—a model already lightly fine-tuned on high-quality examples (LoRA SFT). A system prompt (preamble) explicitly defining NPOV further boosts performance across all tested models.

The SHQ-NPOV Dataset

The study introduces SHQ-NPOV, a specialized dataset designed for low-data regimes. It contains 300 human-written query-answer pairs covering 89 controversial "in-distribution" topics and 30 OOD topics for testing. The dataset was developed through iterative rounds of human peer-critique and expert writer training based on Wikipedia’s NPOV policy.

Examples of queries and answers with their corresponding NPOV scores, illustrating the spectrum from biased to neutral.

Key Results and Insights

The researchers compared PE-RL against several baselines (SFT, LoRA SFT, RLHF, and Best-of-10) using a 20B parameter base model.

  • The "Small Data" Miracle: Remarkably, just 300 high-quality examples were sufficient to train a robust reward model and policy, emphasizing data quality over quantity.
  • Generalization through Constraint: Unlike full-parameter models (SFT/RLHF) which often overfit, PE-RL learned the underlying style of neutrality. By modifying only a tiny fraction of weights, it preserved the model's broad knowledge while successfully steering its "voice."
  • Linguistic Depth: PE-RL moved beyond simple impartiality to provide nuanced, informative responses. Qualitative gains were dramatic: the presence of supportive details increased from 60.25% to 85.21%, and the absence of oversimplification improved from 68.74% to 91.43%.
  • OOD Robustness: PE-RL showed no statistically significant drop in performance when moving to entirely new topics, whereas full SFT models showed clear signs of overfitting.

Conclusion and Future Work

While PE-RL significantly improves the style and neutrality of LLM responses, it does not inherently eliminate hallucinations; models may still generate neutral-sounding falsehoods if they lack factual knowledge. Additionally, the current dataset is US-centric. Future research is required to generalize these findings to diverse cultural contexts and to ensure that all valid human perspectives are sourced and represented fairly.

Summary updated

Abstract

The paper shows that parameter-efficient reinforcement learning (PE-RL) is a highly effective training regime to improve large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e. to provide significantly more informative, diverse and impartial answers. This is shown by evaluating PE-RL and multiple strong baselines-including LoRA finetuning (strongest baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\%\rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating sufficient answers from "great'' answers ($60.25\%\rightarrow 85.21\%$ for presence of supportive details, $68.74\%\rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Moreover, our evaluation also finds a key property of PE-RL for this task: unlike methods that update all parameters, it generalises out of topic. Finally, to enable further studies we also release the dataset, SHQ-NPOV, and provide a methodology to create such datasets through iterative rounds of human peer-critique and annotator training.
BibTeX
@misc{hoffmann2025improving,
  author = {Jessica D. Hoffmann and Christiane Ahlheim and Zac Yu and Aria Walfrand and Jarvis Jin and Marie Tano and Ahmad Beirami and Erin van Liemt and Nithum Thain and Hakim Sidahmed and Lucas Dixon},
  title = {{Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL}},
  year = {2025},
  eprint = {2503.03654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03654},
}
Figure