Abstract
Traditional reward models in Reinforcement Learning from Human Feedback (RLHF) often rely on "naive" signals like community upvotes, which can lead to spurious correlations with answer length or confidence rather than factual accuracy. We present an axiomatic framework for training Preference Models (PMs) that explicitly enforces principles such as truthfulness, groundedness, and relevance. By programmatically generating contrastive training pairs based on six defined axioms, we demonstrate that a specialized 220M-parameter model can outperform significantly larger models like GPT-4 in human preference agreement for longform question answering.
Introduction
The effectiveness of RLHF depends heavily on the quality of the Reward Model (or Preference Model). Existing models often suffer from an "alignment gap," where they struggle to evaluate human-written and LLM-generated answers on a consistent scale. Furthermore, naive models lack explicit guidance on core values, often rewarding "fluff" or confident hallucinations. This paper addresses these issues by introducing a principled, axiomatic approach to data curation and model training.
Methods: The Axiomatic Framework
The core innovation is the decomposition of answer quality into six distinct axioms. Instead of relying solely on raw preference data, the authors use a mix of community-mined data and LLM-generated synthetic pairs to train the model on specific attributes.
The six axioms include:
- Usefulness (Axiom 0): Leverages raw upvote data from Community QA (CQA) sites.
- Relevance (Axiom 1): Uses "hard negatives" by pairing questions with answers to related but distinct queries.
- Groundedness (Axiom 2): Compares open-book (retrieval-augmented) vs. closed-book LLM responses.
- Truthfulness (Axiom 3): Employs LLMs to create factually corrupted versions of high-quality answers.
- Relevant Grounding (Axiom 4): Distinguishes between answers supported by high-quality evidence vs. irrelevant context.
- Thoroughness (Axiom 5): Creates "super-answers" by merging the best points of multiple top-voted human responses.
The model is implemented as a T5-Flan cross-encoder. Training utilizes a margin loss function, sometimes augmented with "Variable Margins" derived from GPT-4 scores to provide a more nuanced training signal.
Evaluation and Results
The model was evaluated across diverse datasets, including Stack Exchange, ELI5, and a newly introduced "Research-Analysis" benchmark consisting of 500 complex, reasoning-heavy questions.
Bridging the Alignment Gap
A key finding is the model's ability to normalize scores across different writing styles. Naive models often penalize LLM-generated answers simply because they differ stylistically from human forum posts. The axiomatic approach successfully bridges this gap by teaching the model to recognize quality features like citations and factual density regardless of the source.
Performance Benchmarks
- Human Agreement: On the Research-Analysis benchmark, the 220M-parameter Axiomatic PM achieved 66.8% agreement with human judges, surpassing GPT-4’s 65.4%.
- Groundedness: The model correctly identified grounded answers 81.3% of the time in zero-shot evaluations.
- Efficiency: Despite its small size, the model outperformed 7B-parameter alternatives like StackLlama and OpenAssistant in ranking tasks across Stack Exchange and ELI5.
Discussion and Limitations
The results suggest that for evaluative tasks, high-quality, principled data is more critical than model scale. However, the current framework is limited to single-turn interactions and produces a single scalar score, which lacks the interpretability of multi-dimensional feedback. Future work will explore multi-turn capabilities and generating per-axiom scores to provide actionable feedback for iterative model alignment.
Summary updated