Introduction
Aligning Large Language Models (LLMs) with human preferences typically relies on Reinforcement Learning from Human Feedback (RLHF). However, the human-in-the-loop requirement presents a significant bottleneck due to high costs and poor scalability. While "Learning from AI Feedback" (LAIF) has emerged as an alternative, it often inherits the instability of reinforcement learning or suffers from mode collapse when trained on purely synthetic data. This paper introduces CITING (Curriculum Instruction TunING), a framework that leverages a powerful "teacher" LLM to guide a "student" LLM through iterative revisions, bypassing reinforcement learning entirely in favor of stable supervised fine-tuning (SFT).
Methods
The CITING framework is inspired by traditional pedagogical methods where a student improves by studying a tutor's corrections against a specific rubric. The process consists of two primary phases:
- Rubric Design: The teacher LLM (e.g., GPT-3.5) analyzes a set of instructions to generate specific evaluation criteria. These rubrics categorize tasks (e.g., "creative writing" vs. "factual retrieval") to provide targeted guidance. A Sentence-BERT model is employed to match new instructions to the most appropriate rubric during training and inference.
- Curriculum Instruction Tuning: The student LLM (e.g., LLaMA-7B) generates an initial response. The teacher then reviews this response alongside the instruction and the relevant rubric to produce a revised, high-quality version. The student is subsequently fine-tuned on the sequence:
[Instruction + Rubric + Initial Response -> Revised Response].
This cycle can be repeated iteratively, allowing the student to learn from increasingly refined "deltas" between its current output and the teacher's ideal response.
Experimental Results
The authors evaluated CITING across several benchmarks, including Alpaca (training distribution), World Knowledge, Reading Comprehension, and Commonsense Reasoning. Performance was assessed by GPT-4 based on three qualitative metrics: Articulation, Depth, and Comprehensiveness.
CITING demonstrated substantial improvements over existing alignment methods. It achieved an average win rate of 79.4% over standard SFT and 73.4% over RLHF. Notably, the model showed strong zero-shot generalization on out-of-distribution tasks. For instance, on World Knowledge tasks, CITING outperformed baselines by over 50% in both articulation and depth.
Qualitative analysis revealed that CITING-trained models naturally adopt more structured response formats and avoid the generic, "hollow" phrasing often associated with standard RLHF outputs.
Discussion and Limitations
A key insight from this work is that generation provides a denser supervisory signal than simple ranking. While RLHF relies on scalar rewards from rankings, CITING utilizes the full linguistic richness of a teacher's revision.
However, the researchers identified a "ceiling" for iterative tuning. Performance gains tend to diminish with each round. Furthermore, excessive iterations (e.g., beyond three rounds) can lead to catastrophic forgetting, where the model begins to lose foundational knowledge learned during initial training. Additionally, while the model excels at refining existing thoughts (improving depth and articulation), it shows more modest gains in "comprehensiveness," as the teacher's revisions typically focus on polishing the student's initial direction rather than introducing entirely new perspectives.
Conclusion
CITING offers a compute-efficient and stable alternative to RLHF by framing alignment as a curriculum-based learning task. By utilizing explicit rubrics and iterative teacher revisions, the framework enables student LLMs to achieve high levels of articulation and reasoning depth without the need for expensive human annotation or complex reinforcement learning pipelines.
Summary updated