Axiomatic Preference Modeling for Longform Question Answering

Corby Rosset, Guoqing Zheng, Victor Dibia, Ahmed Awadallah, Paul N. Bennett

Abstract

Traditional reward models in Reinforcement Learning from Human Feedback (RLHF) often rely on "naive" signals like community upvotes, which can lead to spurious correlations with answer length or confidence rather than factual accuracy. We present an axiomatic framework for training Preference Models (PMs) that explicitly enforces principles such as truthfulness, groundedness, and relevance. By programmatically generating contrastive training pairs based on six defined axioms, we demonstrate that a specialized 220M-parameter model can outperform significantly larger models like GPT-4 in human preference agreement for longform question answering.

Introduction

The effectiveness of RLHF depends heavily on the quality of the Reward Model (or Preference Model). Existing models often suffer from an "alignment gap," where they struggle to evaluate human-written and LLM-generated answers on a consistent scale. Furthermore, naive models lack explicit guidance on core values, often rewarding "fluff" or confident hallucinations. This paper addresses these issues by introducing a principled, axiomatic approach to data curation and model training.

Methods: The Axiomatic Framework

The core innovation is the decomposition of answer quality into six distinct axioms. Instead of relying solely on raw preference data, the authors use a mix of community-mined data and LLM-generated synthetic pairs to train the model on specific attributes.

Overview of the Axiomatic Preference Model training mechanism, illustrating the relationship between defined axioms and their corresponding data generation strategies.

The six axioms include:

  • Usefulness (Axiom 0): Leverages raw upvote data from Community QA (CQA) sites.
  • Relevance (Axiom 1): Uses "hard negatives" by pairing questions with answers to related but distinct queries.
  • Groundedness (Axiom 2): Compares open-book (retrieval-augmented) vs. closed-book LLM responses.
  • Truthfulness (Axiom 3): Employs LLMs to create factually corrupted versions of high-quality answers.
  • Relevant Grounding (Axiom 4): Distinguishes between answers supported by high-quality evidence vs. irrelevant context.
  • Thoroughness (Axiom 5): Creates "super-answers" by merging the best points of multiple top-voted human responses.

The model is implemented as a T5-Flan cross-encoder. Training utilizes a margin loss function, sometimes augmented with "Variable Margins" derived from GPT-4 scores to provide a more nuanced training signal.

Evaluation and Results

The model was evaluated across diverse datasets, including Stack Exchange, ELI5, and a newly introduced "Research-Analysis" benchmark consisting of 500 complex, reasoning-heavy questions.

Bridging the Alignment Gap

A key finding is the model's ability to normalize scores across different writing styles. Naive models often penalize LLM-generated answers simply because they differ stylistically from human forum posts. The axiomatic approach successfully bridges this gap by teaching the model to recognize quality features like citations and factual density regardless of the source.

Comparison of score distributions between Naive and Axiomatic Preference Models. The Axiomatic model significantly reduces the "alignment gap" for ChatGPT answers, bringing their score distribution closer to that of top-voted human answers.

Performance Benchmarks

  • Human Agreement: On the Research-Analysis benchmark, the 220M-parameter Axiomatic PM achieved 66.8% agreement with human judges, surpassing GPT-4’s 65.4%.
  • Groundedness: The model correctly identified grounded answers 81.3% of the time in zero-shot evaluations.
  • Efficiency: Despite its small size, the model outperformed 7B-parameter alternatives like StackLlama and OpenAssistant in ranking tasks across Stack Exchange and ELI5.

Discussion and Limitations

The results suggest that for evaluative tasks, high-quality, principled data is more critical than model scale. However, the current framework is limited to single-turn interactions and produces a single scalar score, which lacks the interpretability of multi-dimensional feedback. Future work will explore multi-turn capabilities and generating per-axiom scores to provide actionable feedback for iterative model alignment.

Summary updated

Abstract

The remarkable abilities of large language models (LLMs) like ChatGPT and GPT-4 partially stem from the post-training processes involving human preferences encoded within a reward model as part of a Reinforcement Learning from Human Feedback (RLHF) regimen. These reward models (RMs) often lack direct knowledge of why, or under what principles, the preferences annotations were made. In this study, we identify principles that guide RMs to better align with human preferences, and then develop an axiomatic framework to generate a rich variety of preference signals to uphold them. We use these axiomatic signals to train a model for the scoring answers to longform questions. Our approach yields a \textbf{Preference Model} with only about 220M parameters that agrees with gold human-annotated preference labels more often than GPT-4. The contributions of this work include: training a standalone preference model that can score human- and LLM-generated answers on the same scale; developing an axiomatic framework for generating training data pairs tailored to certain principles; and showing that a small amount of axiomatic signals can help small models outperform GPT-4 in preference scoring. We intend to release our axiomatic data and model.
BibTeX
@inproceedings{rosset2023axiomatic,
  author = {Corby Rosset and Guoqing Zheng and Victor Dibia and Ahmed Awadallah and Paul N. Bennett},
  title = {{Axiomatic Preference Modeling for Longform Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.702},
}
Figure