Constructive Large Language Model Alignment with Diverse Feedback

Tianshu Yu, Ting-En Lin, Yuchuan Wu, Min Yang, Fei Huang, Yongbin LiSubmission to ICLR 2024 · Withdrawn

TL;DR

Current LLM alignment frameworks rely almost exclusively on a single feedback modality—usually pairwise preferences—which can be data-inefficient and prone to over-optimization. Inspired by Vygotsky’s Zone of Proximal Development (ZPD), Constructive and Diverse Feedback (CDF) splits tasks by difficulty and assigns distinct feedback types: critiques for easy problems, refinements for medium problems, and preference pairs for hard problems. Across question answering, dialog, and summarization, CDF substantially outperforms single-feedback baselines and off-the-shelf models in both RLHF and DPO settings.

What Problem Does This Paper Address?

Aligning large language models (LLMs) with human intent typically involves Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). However, existing pipelines predominantly feed these algorithms a single type of signal: binary pairwise preferences (e.g., "Response A is better than Response B").

This single-feedback paradigm suffers from two major issues:

  1. Inefficiency: It requires vast amounts of preference annotations (often tens or hundreds of thousands of pairs) because simple preference labels provide very sparse learning signals.
  2. One-size-fits-all mismatch: In human learning, a student solving an easy problem benefits from high-level pointers, while an unsolvable problem is better handled by ranking different approaches. Treating every alignment query identically overlooks how difficulty affects the value of feedback.

Methods

CDF designs a differentiated feedback pipeline structured around the educational concept of the Zone of Proximal Development:

                       [ Unaligned Base Model (Vicuna-7B) ]
                                      │
                         Generate initial answer &
                           Calculate Perplexity (PPL)
                                      │
               ┌──────────────────────┼──────────────────────┐
               ▼                      ▼                      ▼
           Low PPL                Medium PPL             High PPL
         [Easy Group]           [Medium Group]         [Hard Group]
               │                      │                      │
       Critique Feedback      Refinement Feedback    Preference Feedback
      (Suggestions -> Model   (API generates full    (API chooses between
         rewrites answer)       revised answer)       Vicuna-7B & 13B)
               │                      │                      │
               └──────────────────────┼──────────────────────┘
                                      ▼
                      Standardized Comparison Pairs (a_w, a_l)
                                      ▼
                           [ RLHF (PPO) or DPO ]
  1. Difficulty Assessment: The base model (Vicuna-7B) generates greedy completions on a problem dataset $D$. The perplexity (PPL) of the completion serves as an uncertainty proxy. Problems are sorted and split into three equal tiers: Easy, Medium, and Hard.
  2. Stratified Feedback Collection (via GPT-3.5-turbo):
  3. Critique (Easy): The model generally knows the facts. The API provides critique/suggestions, and the base model rewrites its own answer.
  4. Refinement (Medium): The problem requires substantial guidance. The API directly rewrites and refines the initial answer into a high-quality version.
  5. Preference (Hard): The base model cannot easily solve or refine the task. An alternative answer is sampled from a larger model (Vicuna-13B), and the API selects the preferred completion.
  6. Training: All three feedback streams are converted into standard preference pairs $(a_w, a_l)$ and trained via either PPO-based RLHF or DPO.

Datasets

The authors constructed a combined training dataset $D$ of 30,000 samples by sampling 10,000 examples from each of three widely used alignment benchmarks:

  • WebGPT-comparison (Question Answering)
  • HH-RLHF (Dialogue Generation)
  • OpenAI-Summarize-TLDR (Text Summarization)

For RLHF policy training, an additional 30,000 prompts were collected from sources including ELI5, TriviaQA, and ARC. Evaluation used a dedicated test split of 300 held-out prompts (100 per task).

Key Insights and Ideas

  • Model Self-Correction Has Limits: Self-correction via natural language critiques works well when the model already understands the task (low perplexity). But on difficult prompts, self-correction falters—making direct external rewrites or ranked alternatives far more valuable.
  • Different Feedback Reduces Over-Optimization: In standard RLHF, reward models trained on a single feedback type quickly suffer from over-optimization (the policy exploits reward proxies, causing actual response quality to drop). Reward models trained on CDF’s mixed feedback exhibited higher validation accuracy (88.51% vs. 76.00%–87.45%) and sustained quality gains across PPO training steps.
  • Data Efficiency via Information Density: Richer instructional feedback (critiques and edits) allows the model to learn efficiently on simpler examples, reserving expensive ranking/comparison data for truly hard queries.

Key Results

Experiments evaluated CDF against base Vicuna-7B, LLaMa-2-7B-Chat, the original human-annotated "Golden" responses, and single-feedback baselines using GPT-4 scoring (1–5 scale), the OpenAssistant Reward Model (RM), and human pairwise evaluation:

Model GPT-4 Avg RM Avg
Vicuna-7B (Base) 3.49 0.609
LLaMa-2-7B-Chat 3.65 0.814
Golden Responses 2.83 -0.492
RLHF (Best Single Feedback: Prefer) 3.82 0.951
CDF-RLHF (Ours) 4.10 1.112
DPO (Best Single Feedback: Refine) 3.80 0.915
CDF-DPO (Ours) 4.37 1.389
  • Win Rates: In head-to-head human evaluations, CDF-DPO beat the original dataset's golden responses with a 78.2% win rate (vs. 11.6% loss) and beat its strongest single-feedback baseline (DPO_Refine) with a 49.5% win rate (vs. 14.5% loss, 36.0% tie).
  • Across Tasks: CDF provided the largest relative margin on summarization, where the base model had received less initial instruction tuning.

Limitations and Future Work

  • Simulated vs. Real Human Feedback: The authors rely on GPT-3.5-turbo as an affordable proxy annotator; investigating the effects of genuine human crowdworkers within this stratified framework remains for future study.
  • Perplexity as a Difficulty Heuristic: Token perplexity measures language modeling uncertainty, but it does not always correlate with logical or reasoning difficulty.
  • Evaluation Scale: The test suite consists of 300 prompts across three tasks; broader evaluations on safety, coding, and multi-turn reasoning benchmarks could test the limits of the framework.

Summary updated

Abstract

In recent research on large language models (LLMs), there has been a growing emphasis on aligning these models with human values to reduce the impact of harmful content. However, current alignment methods often rely solely on singular forms of human feedback, such as preferences, annotated labels, or natural language critiques, overlooking the potential advantages of combining these feedback types. This limitation leads to suboptimal performance, even when ample training data is available. In this paper, we introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance LLM alignment. Our approach involves collecting three distinct types of feedback tailored to problems of varying difficulty levels within the training dataset. Specifically, we exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data. To assess the effectiveness of CDF, we evaluate it against previous methods in three downstream tasks: question answering, dialog generation, and text summarization. Experimental results demonstrate that CDF achieves superior performance even with a smaller training dataset.
BibTeX
@misc{yu2023constructive,
  author = {Tianshu Yu and Ting-En Lin and Yuchuan Wu and Min Yang and Fei Huang and Yongbin Li},
  title = {{Constructive Large Language Model Alignment with Diverse Feedback}},
  year = {2023},
  eprint = {2310.06450},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06450},
}

Loading related content...

Figure