TL;DR
Training large language models typically relies on expensive human alignment or synthetic data generated from scratch, which often suffers from hallucinations and mode collapse. CITING (Curriculum Instruction TunING) introduces a tutor-student paradigm where a teacher LLM establishes explicit evaluation rubrics and iteratively rewrites the student LLM’s responses. By fine-tuning the student on these rubric-guided revisions, CITING outperforms standard SFT and RLHF baselines with an average win rate of over 73%.
What Problem Does This Paper Address?
Instruction tuning and Reinforcement Learning from Human Feedback (RLHF) have substantially improved language model outputs, but scaling them is bottle-necked by the time and expense of human annotation.
While recent efforts focus on "Learning from AI Feedback" (LAIF), most borrow the human framework of ranking candidate outputs using reinforcement learning. However, humans prefer ranking simply because drafting high-quality text from scratch is cognitively demanding—a constraint that LLMs do not share. Furthermore, generating purely synthetic data from scratch frequently induces model collapse, hallucination, and high reinforcement learning instability.
The authors investigate how to better leverage the generative capacity of a teacher LLM to provide targeted, supervised feedback without requiring complex RL pipelines.
Methods
CITING mimics the dynamic between a human tutor and a student by breaking the alignment process into rubric generation and iterative revision fine-tuning:
- Teacher Rubric Design:
- A teacher LLM (GPT-4) categorizes instructions into five general archetypes (Factual Knowledge, Explanation/Definition, Analysis/Evaluation, Creative Generation, Practical Application) and defines explicit quality criteria for each.
- At test time or for new queries, a Sentence-BERT module assigns the most relevant rubric using cosine similarity matching.
- Supervised Fine-Tuning (SFT):
- A student LLM (LLaMA-7B) is initialized via standard supervised fine-tuning.
- Curriculum Instruction Tuning via Revision:
- Given an instruction and its assigned rubric, the student model generates an initial output.
- A teacher model (GPT-3.5) revises the student’s output according to the rubric, providing a concrete correction.
- The student is then fine-tuned on the prompt containing the instruction, rubric, and its original response, learning to predict the revised output.
- Iterative Inference and Training:
- The process can be repeated over multiple rounds ($N$), training the student to iteratively critique and polish its own responses.
Instruction + Rubric + Student Response (r_k)
│
▼
[Teacher LLM Revision]
│
▼
Targeted Supervision (r_k+1) ──► Student Fine-Tuning
Datasets
- Training & In-Distribution: Stanford Alpaca (52,000 instructions generated by
text-davinci-003; 1,000 sampled instruction–revision pairs were used for the curriculum tuning stage). - Zero-Shot Evaluation Benchmarks:
- World Knowledge: Combined NaturalQuestions and TriviaQA.
- Reading Comprehension: Curated mixture of SQuAD, QuAC, and BoolQ.
- Commonsense Reasoning: Curated mixture of PIQA, OpenBookQA, and CommonsenseQA.
Key Insights and Ideas
- Revising vs. Ranking: In LAIF, forcing a teacher LLM to produce scalar rewards or rankings wastes its best ability: rich, natural-language generation. Having the teacher directly edit the student’s specific flaws produces higher-density training signal.
- Curriculum Through Rubrics: Explicitly conditioning the revision process on structured rubrics helps the model learn standardized principles of response quality (e.g., depth, clarity, and conciseness) instead of mimicking arbitrary phrasing.
- Generalization Over Memorization: Performance improvements were substantially higher on out-of-distribution zero-shot reasoning tasks than on in-distribution Alpaca tasks, proving the student learned how to reason and polish rather than just memorizing answers.
Key Results
Evaluated by GPT-4 across three qualitative dimensions (Articulate, In-depth, and Comprehensive), CITING significantly outperformed all baselines:
- Overall Head-to-Head Win Rates:
- 79.4% against Supervised Fine-Tuning (SFT).
- 73.4% against RLHF (implemented via
trlx). - 78.1% against RRHF (Rank Responses to align with Human Feedback).
- 76.3% against RAFT (Reward Ranked Fine-Tuning).
- Dimensional Gains: CITING achieved its largest performance edge in In-depth analysis (often scoring 80%+ win rates on reasoning tasks), showing that iterative critiques train the model to unpack details rather than staying surface-level.
Limitations and Future Work
- Catastrophic Forgetting in Later Rounds: Iterative curriculum fine-tuning exhibits diminishing returns. By the fourth round ($k=4$), model performance drops on certain metrics (such as Reading Comprehension), likely due to the model overwriting foundational knowledge acquired during initial SFT.
- Limited Breadth: CITING showed its smallest gains on the Comprehensive metric. Because the teacher only refines existing responses rather than introducing external facts or broad sub-topics, the student does not inherently learn to expand topical coverage.
- Rubric Classification Errors: The Sentence-BERT similarity matching pipeline achieved a ~72% rubric assignment accuracy, suggesting room for tighter instruction-to-rubric mapping.
Summary updated