CITING: Large Language Models Create Curriculum for Instruction Tuning

Tao Feng, Zifeng Wang, Jimeng SunSubmission to ICLR 2024 · Rejected

TL;DR

Training large language models typically relies on expensive human alignment or synthetic data generated from scratch, which often suffers from hallucinations and mode collapse. CITING (Curriculum Instruction TunING) introduces a tutor-student paradigm where a teacher LLM establishes explicit evaluation rubrics and iteratively rewrites the student LLM’s responses. By fine-tuning the student on these rubric-guided revisions, CITING outperforms standard SFT and RLHF baselines with an average win rate of over 73%.

What Problem Does This Paper Address?

Instruction tuning and Reinforcement Learning from Human Feedback (RLHF) have substantially improved language model outputs, but scaling them is bottle-necked by the time and expense of human annotation.

While recent efforts focus on "Learning from AI Feedback" (LAIF), most borrow the human framework of ranking candidate outputs using reinforcement learning. However, humans prefer ranking simply because drafting high-quality text from scratch is cognitively demanding—a constraint that LLMs do not share. Furthermore, generating purely synthetic data from scratch frequently induces model collapse, hallucination, and high reinforcement learning instability.

The authors investigate how to better leverage the generative capacity of a teacher LLM to provide targeted, supervised feedback without requiring complex RL pipelines.

Methods

CITING mimics the dynamic between a human tutor and a student by breaking the alignment process into rubric generation and iterative revision fine-tuning:

  1. Teacher Rubric Design:
  2. A teacher LLM (GPT-4) categorizes instructions into five general archetypes (Factual Knowledge, Explanation/Definition, Analysis/Evaluation, Creative Generation, Practical Application) and defines explicit quality criteria for each.
  3. At test time or for new queries, a Sentence-BERT module assigns the most relevant rubric using cosine similarity matching.
  4. Supervised Fine-Tuning (SFT):
  5. A student LLM (LLaMA-7B) is initialized via standard supervised fine-tuning.
  6. Curriculum Instruction Tuning via Revision:
  7. Given an instruction and its assigned rubric, the student model generates an initial output.
  8. A teacher model (GPT-3.5) revises the student’s output according to the rubric, providing a concrete correction.
  9. The student is then fine-tuned on the prompt containing the instruction, rubric, and its original response, learning to predict the revised output.
  10. Iterative Inference and Training:
  11. The process can be repeated over multiple rounds ($N$), training the student to iteratively critique and polish its own responses.
Instruction + Rubric + Student Response (r_k) 
             │
             ▼
   [Teacher LLM Revision]
             │
             ▼
Targeted Supervision (r_k+1) ──► Student Fine-Tuning

Datasets

  • Training & In-Distribution: Stanford Alpaca (52,000 instructions generated by text-davinci-003; 1,000 sampled instruction–revision pairs were used for the curriculum tuning stage).
  • Zero-Shot Evaluation Benchmarks:
  • World Knowledge: Combined NaturalQuestions and TriviaQA.
  • Reading Comprehension: Curated mixture of SQuAD, QuAC, and BoolQ.
  • Commonsense Reasoning: Curated mixture of PIQA, OpenBookQA, and CommonsenseQA.

Key Insights and Ideas

  • Revising vs. Ranking: In LAIF, forcing a teacher LLM to produce scalar rewards or rankings wastes its best ability: rich, natural-language generation. Having the teacher directly edit the student’s specific flaws produces higher-density training signal.
  • Curriculum Through Rubrics: Explicitly conditioning the revision process on structured rubrics helps the model learn standardized principles of response quality (e.g., depth, clarity, and conciseness) instead of mimicking arbitrary phrasing.
  • Generalization Over Memorization: Performance improvements were substantially higher on out-of-distribution zero-shot reasoning tasks than on in-distribution Alpaca tasks, proving the student learned how to reason and polish rather than just memorizing answers.

Key Results

Evaluated by GPT-4 across three qualitative dimensions (Articulate, In-depth, and Comprehensive), CITING significantly outperformed all baselines:

  • Overall Head-to-Head Win Rates:
  • 79.4% against Supervised Fine-Tuning (SFT).
  • 73.4% against RLHF (implemented via trlx).
  • 78.1% against RRHF (Rank Responses to align with Human Feedback).
  • 76.3% against RAFT (Reward Ranked Fine-Tuning).
  • Dimensional Gains: CITING achieved its largest performance edge in In-depth analysis (often scoring 80%+ win rates on reasoning tasks), showing that iterative critiques train the model to unpack details rather than staying surface-level.

Limitations and Future Work

  • Catastrophic Forgetting in Later Rounds: Iterative curriculum fine-tuning exhibits diminishing returns. By the fourth round ($k=4$), model performance drops on certain metrics (such as Reading Comprehension), likely due to the model overwriting foundational knowledge acquired during initial SFT.
  • Limited Breadth: CITING showed its smallest gains on the Comprehensive metric. Because the teacher only refines existing responses rather than introducing external facts or broad sub-topics, the student does not inherently learn to expand topical coverage.
  • Rubric Classification Errors: The Sentence-BERT similarity matching pipeline achieved a ~72% rubric assignment accuracy, suggesting room for tighter instruction-to-rubric mapping.

Summary updated

Abstract

The recent advancement of large language models (LLMs) has been achieved through a combo of instruction tuning and human alignment. However, building manually crafted instruction datasets and performing human alignment become the bottleneck for scaling the development of LLMs. In this paper, we exploit the idea of leveraging AI models in lieu of humans as the teacher to train student LLMs. Our method is inspired by how human students refine their writing skills by following the rubrics and learning from the revisions offered by their tutors. Specifically, we employ a teacher LLM to create a curriculum for instruction tuning of the student LLM, namely Curriculum Instruction TunING (CITING). It encompasses two main steps: (1) the teacher LLM crafts the rubrics for evaluating the answers corresponding to various types of questions, and (2) the student LLM learns to follow the rubrics and perform self-correction from the revision made by the teacher. We further iteratively carry out it to embody the procedure of CITING. We compare CITING to a series of state-of-the-art baselines on four datasets. Our method demonstrates strong improvement in terms of articulate, in-depth, and comprehensive by GPT-4 evaluation. Specifically, it achieves an average winning rate of 79.4\% over SFT, 73.4\% over RLHF, 78.1\% over RRHF, and 76.3\% over RAFT, respectively.
BibTeX
@misc{feng2023citing,
  author = {Tao Feng and Zifeng Wang and Jimeng Sun},
  title = {{CITING: Large Language Models Create Curriculum for Instruction Tuning}},
  year = {2023},
  eprint = {2310.02527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.02527},
}
Figure