Abstract
In recent research on large language models (LLMs), there has been a growing emphasis on aligning these models with human values to reduce the impact of harmful content. However, current alignment methods often rely solely on singular forms of human feedback, such as preferences, annotated labels, or natural language critiques, overlooking the potential advantages of combining these feedback types. This limitation leads to suboptimal performance, even when ample training data is available. In this paper, we introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance LLM alignment, inspired by constructivist learning theory. Our approach involves collecting three distinct types of feedback tailored to problems of varying difficulty levels within the training dataset. Specifically, we exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data. To assess the effectiveness of CDF, we evaluate it against previous methods in three downstream tasks: question answering, dialog generation, and text summarization. Experimental results demonstrate that CDF achieves superior performance even with a smaller training dataset.
Introduction
Aligning Large Language Models (LLMs) with human values—ensuring they are helpful, honest, and harmless—is a critical challenge in AI development. The standard approach, Reinforcement Learning from Human Feedback (RLHF), typically relies on a "data bottleneck" of preference pairs (e.g., "Response A is better than Response B"). This binary comparison often fails to capture the nuance required for complex tasks or provides redundant information for simple ones.
Drawing inspiration from the Zone of Proximal Development (ZPD)—a constructivist learning theory suggesting that students learn best when instruction is tailored to their current ability—we propose Constructive and Diverse Feedback (CDF). This framework treats the LLM as a student, providing differentiated feedback based on the inherent difficulty of the task at hand.
The CDF Framework
The CDF methodology follows a structured three-step process designed to maximize information density during the alignment phase.
1. Difficulty Assessment
First, an unaligned base model (e.g., Vicuna-7B) generates initial responses to a problem dataset. The difficulty of each problem is quantified using perplexity (PPL). High perplexity indicates that the model is "surprised" by the sequence, serving as a proxy for task difficulty. Problems are then sorted into three categories: Easy, Medium, and Hard.
2. Differentiated Feedback Collection
Instead of applying a uniform feedback mechanism, CDF matches the feedback type to the difficulty tier:
- Easy (Critique Feedback): For simple tasks, a "teacher" model (GPT-3.5) provides a natural language critique. The student model then uses this critique to generate an improved answer.
- Medium (Refinement Feedback): For tasks of moderate difficulty, the teacher model directly provides a refined, high-quality version of the answer.
- Hard (Preference Feedback): For complex tasks where self-correction is insufficient, the student model's output is compared against a stronger model (Vicuna-13B), and the teacher model selects the superior response.
3. Model Training
The resulting dataset of diverse feedback is used to fine-tune the model. The researchers evaluated both standard RLHF and Direct Preference Optimization (DPO), the latter being a more stable method that optimizes the policy directly on preference pairs without an explicit reward model.
Experimental Results
The CDF framework was tested across three major benchmarks: WebGPT-comparison (QA), HH-RLHF (Dialogue), and OpenAI-Summarize-TLDR (Summarization).
Superior Alignment Performance
As shown in the evaluation results, the CDF-DPO model consistently outperformed established baselines, including the base Vicuna-7B and the instruction-tuned Llama-2-7B-Chat. Notably, the CDF-trained models achieved higher win rates against "Golden" (human-annotated) responses, suggesting that structured, simulated feedback can be more instructive than noisy human labels.
Efficiency and Robustness
One of the most significant findings is that CDF achieves these results with a relatively small dataset of 30,000 samples. By providing more "constructive" information (critiques and refinements) for simpler tasks, the model learns more efficiently than it would from binary preferences alone. Furthermore, the diversity of the feedback helps mitigate "reward hacking," a common issue in RLHF where models learn to exploit the reward function rather than truly improving their helpfulness.
Conclusion
Constructive and Diverse Feedback (CDF) represents a shift from treating LLM alignment as a simple statistical optimization to treating it as a pedagogical process. By tailoring feedback to task difficulty, CDF reduces the data burden of alignment while producing models that are more closely aligned with human intent. Future work will explore the scalability of this approach to larger model architectures and the refinement of perplexity as a difficulty metric.
Summary updated