CITING: Large Language Models Create Curriculum for Instruction Tuning

Tao Feng, Zifeng Wang, Jimeng Sun

Introduction

Aligning Large Language Models (LLMs) with human preferences typically relies on Reinforcement Learning from Human Feedback (RLHF). However, the human-in-the-loop requirement presents a significant bottleneck due to high costs and poor scalability. While "Learning from AI Feedback" (LAIF) has emerged as an alternative, it often inherits the instability of reinforcement learning or suffers from mode collapse when trained on purely synthetic data. This paper introduces CITING (Curriculum Instruction TunING), a framework that leverages a powerful "teacher" LLM to guide a "student" LLM through iterative revisions, bypassing reinforcement learning entirely in favor of stable supervised fine-tuning (SFT).

Methods

The CITING framework is inspired by traditional pedagogical methods where a student improves by studying a tutor's corrections against a specific rubric. The process consists of two primary phases:

  1. Rubric Design: The teacher LLM (e.g., GPT-3.5) analyzes a set of instructions to generate specific evaluation criteria. These rubrics categorize tasks (e.g., "creative writing" vs. "factual retrieval") to provide targeted guidance. A Sentence-BERT model is employed to match new instructions to the most appropriate rubric during training and inference.
  2. Curriculum Instruction Tuning: The student LLM (e.g., LLaMA-7B) generates an initial response. The teacher then reviews this response alongside the instruction and the relevant rubric to produce a revised, high-quality version. The student is subsequently fine-tuned on the sequence: [Instruction + Rubric + Initial Response -> Revised Response].
The CITING framework overview, including rubric design and the iterative tuning process.

This cycle can be repeated iteratively, allowing the student to learn from increasingly refined "deltas" between its current output and the teacher's ideal response.

Experimental Results

The authors evaluated CITING across several benchmarks, including Alpaca (training distribution), World Knowledge, Reading Comprehension, and Commonsense Reasoning. Performance was assessed by GPT-4 based on three qualitative metrics: Articulation, Depth, and Comprehensiveness.

CITING demonstrated substantial improvements over existing alignment methods. It achieved an average win rate of 79.4% over standard SFT and 73.4% over RLHF. Notably, the model showed strong zero-shot generalization on out-of-distribution tasks. For instance, on World Knowledge tasks, CITING outperformed baselines by over 50% in both articulation and depth.

Win rates of CITING compared to RLHF across various evaluation datasets.

Qualitative analysis revealed that CITING-trained models naturally adopt more structured response formats and avoid the generic, "hollow" phrasing often associated with standard RLHF outputs.

Discussion and Limitations

A key insight from this work is that generation provides a denser supervisory signal than simple ranking. While RLHF relies on scalar rewards from rankings, CITING utilizes the full linguistic richness of a teacher's revision.

However, the researchers identified a "ceiling" for iterative tuning. Performance gains tend to diminish with each round. Furthermore, excessive iterations (e.g., beyond three rounds) can lead to catastrophic forgetting, where the model begins to lose foundational knowledge learned during initial training. Additionally, while the model excels at refining existing thoughts (improving depth and articulation), it shows more modest gains in "comprehensiveness," as the teacher's revisions typically focus on polishing the student's initial direction rather than introducing entirely new perspectives.

Conclusion

CITING offers a compute-efficient and stable alternative to RLHF by framing alignment as a curriculum-based learning task. By utilizing explicit rubrics and iterative teacher revisions, the framework enables student LLMs to achieve high levels of articulation and reasoning depth without the need for expensive human annotation or complex reinforcement learning pipelines.

Summary updated

Abstract

The recent advancement of large language models (LLMs) has been achieved through a combo of instruction tuning and human alignment. However, building manually crafted instruction datasets and performing human alignment become the bottleneck for scaling the development of LLMs. In this paper, we exploit the idea of leveraging AI models in lieu of humans as the teacher to train student LLMs. Our method is inspired by how human students refine their writing skills by following the rubrics and learning from the revisions offered by their tutors. Specifically, we employ a teacher LLM to create a curriculum for instruction tuning of the student LLM, namely Curriculum Instruction TunING (CITING). It encompasses two main steps: (1) the teacher LLM crafts the rubrics for evaluating the answers corresponding to various types of questions, and (2) the student LLM learns to follow the rubrics and perform self-correction from the revision made by the teacher. We further iteratively carry out it to embody the procedure of CITING. We compare CITING to a series of state-of-the-art baselines on four datasets. Our method demonstrates strong improvement in terms of articulate, in-depth, and comprehensive by GPT-4 evaluation. Specifically, it achieves an average winning rate of 79.4\% over SFT, 73.4\% over RLHF, 78.1\% over RRHF, and 76.3\% over RAFT, respectively.
BibTeX
@misc{feng2023citing,
  author = {Tao Feng and Zifeng Wang and Jimeng Sun},
  title = {{CITING: Large Language Models Create Curriculum for Instruction Tuning}},
  year = {2023},
  eprint = {2310.02527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.02527},
}
Figure