Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon

TL;DR

This paper demonstrates that Parameter-Efficient Reinforcement Learning (PE-RL) is an incredibly data-efficient way to train large language models (LLMs) to answer sensitive, controversial questions neutrally. Using a tiny dataset of just 300 examples, the authors trained a model that not only provides balanced, multi-perspective answers but also generalizes perfectly to entirely unseen topics without overfitting.

What Problem Does This Paper Address?

When users ask LLMs about controversial topics (e.g., political debates, ethical dilemmas, or alternative medicine), the models often fail to provide balanced responses. They might hallucinate, take a biased stance, or oversimplify complex arguments.

While adapting Wikipedia’s Neutral Point of View (NPOV) policy seems like a natural solution, training LLMs to generate high-quality, multi-perspective text typically requires massive, expensive datasets. The authors tackle a dual challenge: how to effectively guide LLMs toward generating nuanced, impartial text, and how to do it without the prohibitive cost of large-scale data collection.

Methods

The core contribution of the paper is identifying Parameter-Efficient Reinforcement Learning (PE-RL) as a highly effective paradigm for low-data controlled generation.

Instead of updating all billions of parameters in a 20B-parameter base LLM (which is computationally expensive and prone to overfitting), the authors use Low-Rank Adaptation (LoRA) adapters. The pipeline consists of three main steps:

  1. Reward Model Training: A rank-4 LoRA adapter is trained as a pointwise reward model using the 300 annotated examples.
  2. Policy Initialization: The policy model is initialized using a checkpoint trained via LoRA Supervised Fine-Tuning (SFT) on the highest-quality subset of the data.
  3. RL Optimization: The policy's LoRA adapter parameters are optimized via reinforcement learning using the trained reward model.
Figure 1: Pipeline to create Neutral Point of View (NPOV) answers to queries on sensitive topics.

Datasets

The authors introduced a new dataset called SHQ-NPOV:

  • Training/Reward Modeling Set: 300 human-written query-answer pairs covering 89 controversial, US-centric topics. Each pair was evaluated by 3–5 expert and crowd annotators on a 0–5 NPOV scale.
  • Evaluation Set: 1,190 queries generated using 10 distinct prompt templates. This includes the 89 in-distribution topics and 30 entirely out-of-distribution (OOD) topics to test the model's generalization limits.

Key Insights and Ideas

  • Remarkable Data Efficiency: Traditional RLHF typically requires thousands of examples. PE-RL succeeds with only 300 examples because it allows the reward model to leverage negative (highly opinionated or biased) examples, which are often much easier to source or write than perfectly neutral ones.
  • Immunity to Overfitting: Traditional full-parameter tuning methods (SFT and RLHF) suffer from severe overfitting in low-data regimes, causing performance to plummet when tested on new topics. Because PE-RL restricts parameter updates to low-rank adapters, it exhibits no perceptible overfitting.
  • Beyond Simple Impartiality: Through collaboration with expert linguists, the authors identified that "great" neutral answers don't just avoid taking sides; they must actively include Supportive Details (facts, context) and avoid Oversimplification (reducing complex issues to simple dichotomies).
Fractions of examples with Supportive Details and Oversimplification by NPOV score

Key Results

The combination of PE-RL, LoRA SFT initialization, and a prompt preamble (PE-RL + LoRA SFT + preamble) outperformed all baselines, including full-parameter SFT and traditional RLHF:

  • Overall NPOV Quality: Achieved an NPOV classification rate of 98.99% on in-distribution topics and 99.33% on out-of-distribution topics, beating the strongest baseline (LoRA SFT).
  • Linguistic Nuance: Drastically improved the generation of "great" answers. The presence of Supportive Details jumped from 60.25% (LoRA SFT) to 85.21%, while instances of Oversimplification plummeted from 68.74% to just 8.31%.
  • Flawless Generalization: As shown in the quantitative results, there was no statistically significant performance drop between in-distribution and out-of-distribution topics, proving excellent zero-shot domain generalization.

Limitations and Future Work

Despite its success, the authors note several limitations:

  • Hallucinations: Parameter-efficient tuning does not inherently fix the underlying LLM's tendency to hallucinate facts, meaning PE-RL must be paired with separate hallucination mitigation techniques (like Retrieval-Augmented Generation).
  • Cultural Bias: The dataset and definition of neutrality are strictly US-centric. Defining what constitutes "neutrality" globally or for specific sub-populations remains an open, complex societal question.
  • Sourcing Perspectives: The paper assumes the perspectives are already known; automated methods for appropriately sourcing diverse human viewpoints still need further research.

Summary updated

Abstract

The paper shows that parameter-efficient reinforcement learning (PE-RL) is a highly effective training regime to improve large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e. to provide significantly more informative, diverse and impartial answers. This is shown by evaluating PE-RL and multiple strong baselines-including LoRA finetuning (strongest baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\%\rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating sufficient answers from "great'' answers ($60.25\%\rightarrow 85.21\%$ for presence of supportive details, $68.74\%\rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Moreover, our evaluation also finds a key property of PE-RL for this task: unlike methods that update all parameters, it generalises out of topic. Finally, to enable further studies we also release the dataset, SHQ-NPOV, and provide a methodology to create such datasets through iterative rounds of human peer-critique and annotator training.
BibTeX
@misc{hoffmann2025improving,
  author = {Jessica Hoffmann and Christiane Ahlheim and Zac Yu and Aria Walfrand and Jarvis Jin and Marie Tano and Ahmad Beirami and Erin van Liemt and Nithum Thain and Hakim Sidahmed and Lucas Dixon},
  title = {{Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL}},
  year = {2025},
  eprint = {2503.03654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03654},
}
Figure