Abstract
The paper shows that parameter-efficient reinforcement learning (PE-RL) is a highly effective training regime to improve large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e. to provide significantly more informative, diverse and impartial answers. This is shown by evaluating PE-RL and multiple strong baselines—including LoRA finetuning (strongest baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06\% \rightarrow 99.08\%$), but also scores much higher on features linguists identify as key to separating sufficient answers from "great" answers ($60.25\% \rightarrow 85.21\%$ for presence of supportive details, $68.74\% \rightarrow 91.43\%$ for absence of oversimplification). A qualitative analysis corroborates this. Moreover, our evaluation also finds a key property of PE-RL for this task: unlike methods that update all parameters, it generalizes out of topic. Finally, to enable further studies we also release the dataset, SHQ-NPOV, and provide a methodology to create such datasets through iterative rounds of human peer-critique and annotator training.
Introduction
As Large Language Models (LLMs) increasingly serve as primary information interfaces, their tendency to adopt specific biases or oversimplify complex, sensitive issues poses a significant risk of exacerbating social polarization. Teaching models to maintain a Neutral Point of View (NPOV) is challenging due to the scarcity of high-quality training data and the tendency of standard fine-tuning methods to overfit to specific training topics, failing to generalize to new, "out-of-distribution" (OOD) subjects.
Methodology: Parameter-Efficient Reinforcement Learning
To address these challenges, researchers from Google DeepMind propose a pipeline centered on Parameter-Efficient Reinforcement Learning (PE-RL) using LoRA (Low-Rank Adaptation).
Instead of updating billions of parameters, the method trains two small LoRA adapters: one for a Reward Model (to score neutrality) and one for the Policy (the LLM generating text). The best results are achieved by initializing the RL policy with a "warm" checkpoint—a model already lightly fine-tuned on high-quality examples (LoRA SFT). A system prompt (preamble) explicitly defining NPOV further boosts performance across all tested models.
The SHQ-NPOV Dataset
The study introduces SHQ-NPOV, a specialized dataset designed for low-data regimes. It contains 300 human-written query-answer pairs covering 89 controversial "in-distribution" topics and 30 OOD topics for testing. The dataset was developed through iterative rounds of human peer-critique and expert writer training based on Wikipedia’s NPOV policy.
Key Results and Insights
The researchers compared PE-RL against several baselines (SFT, LoRA SFT, RLHF, and Best-of-10) using a 20B parameter base model.
- The "Small Data" Miracle: Remarkably, just 300 high-quality examples were sufficient to train a robust reward model and policy, emphasizing data quality over quantity.
- Generalization through Constraint: Unlike full-parameter models (SFT/RLHF) which often overfit, PE-RL learned the underlying style of neutrality. By modifying only a tiny fraction of weights, it preserved the model's broad knowledge while successfully steering its "voice."
- Linguistic Depth: PE-RL moved beyond simple impartiality to provide nuanced, informative responses. Qualitative gains were dramatic: the presence of supportive details increased from 60.25% to 85.21%, and the absence of oversimplification improved from 68.74% to 91.43%.
- OOD Robustness: PE-RL showed no statistically significant drop in performance when moving to entirely new topics, whereas full SFT models showed clear signs of overfitting.
Conclusion and Future Work
While PE-RL significantly improves the style and neutrality of LLM responses, it does not inherently eliminate hallucinations; models may still generate neutral-sounding falsehoods if they lack factual knowledge. Additionally, the current dataset is US-centric. Future research is required to generalize these findings to diverse cultural contexts and to ensure that all valid human perspectives are sourced and represented fairly.
Summary updated