SCoder: Progressive Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs

Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, Mengdi ZhangEMNLP 2025

TL;DR

This paper introduces SCoder, a novel method to train small, open-source LLMs to generate high-quality code instruction data. By using an iterative self-distillation process, SCoder bootstraps these smaller models into powerful synthesizers, significantly reducing the reliance on expensive, proprietary LLMs and lowering the cost of creating instruction datasets for code generation.

What Problem Does This Paper Address?

Training state-of-the-art code large language models (LLMs) typically requires vast amounts of high-quality instruction data. The common practice is to distill this data from powerful, proprietary LLMs like GPT-3.5 and GPT-4. However, this approach is prohibitively expensive due to the high cost of API calls. The paper addresses this challenge by exploring whether smaller, open-source LLMs can be leveraged to generate this crucial instruction data, thereby democratizing access to powerful code LLMs and reducing development costs.

Methods

The core of SCoder is an iterative self-distillation bootstrap method. This process starts with a small LLM that has been "enhanced" by a limited set of proprietary LLM-distilled samples. In each iteration:

  1. Multi-Checkpoint Sampling & Multi-Aspect Scoring: The current synthesizer generates multiple candidate code instruction pairs from various checkpoints. These candidates are then evaluated by a multi-aspect scorer that assesses them across several criteria (e.g., consistency, relevance, difficulty, code correctness) to ensure diversity and reliability.
  2. Gradient-based Influence Estimation: To further refine the selection, the method uses a gradient-based approach. It compares the gradients of the self-distilled samples with those of proprietary LLM-distilled samples. This helps identify samples that are most influential in improving model performance, mimicking the impact of high-quality proprietary data.
  3. Synthesizer Training: The selected high-quality, influential self-distilled samples are then used to further train the synthesizer itself, iteratively improving its data synthesis capabilities.

This process allows the small LLM to progressively become a more powerful data synthesizer without requiring additional calls to proprietary LLMs after the initial bootstrapping phase.

Datasets

  • Training Data for Synthesizers:
    • A limited set of proprietary LLM-distilled samples (e.g., 10K from GPT-40).
  • Self-distilled data generated iteratively by the bootstrapped synthesizers (20K, 40K, and 60K samples across iterations).
  • Code Instruction Data for SCoder Models:
    • 60K synthesized data from bootstrapped small-scale synthesizers.
  • Evol-codealpaca-v1 dataset (approx. 110K samples) for initial fine-tuning of SCoder.
  • Evaluation Benchmarks:
    • HumanEval (+)
    • MBPP (+)
    • LiveCodeBench (V4)
    • BigCodeBench

Key Insights and Ideas

  • Small LLMs Can Be Powerful Synthesizers: The paper demonstrates that even relatively small open-source LLMs can be trained to generate high-quality code instruction data, challenging the notion that only massive proprietary models can achieve this.
  • Iterative Self-Distillation is Key: The progressive nature of the self-distillation process is crucial. Each iteration builds upon the previous one, allowing the synthesizer to learn from its own outputs and continuously improve its synthesis quality.
  • Gradient Influence is a Strong Signal: Using gradient similarity to proprietary data as a filtering mechanism is an effective way to identify samples that are likely to contribute most to model performance, even without direct access to proprietary LLMs during the main synthesis phase.
  • Cost-Effectiveness: The method offers a significant cost reduction compared to traditional distillation from proprietary LLMs, making advanced code LLM development more accessible.

Key Results

  • Synthesizer Improvement: Bootstrapped synthesizers (e.g., Qwen2.5-Coder-7B/14B-Ins) showed significant improvements in data synthesis capability after just two iterations, leading to better performance when their generated data was used to fine-tune code generation models. For instance, data synthesized by Llama3.1-8B-Ins after two iterations improved HumanEval performance by 5.0% and MBPP by 5.9%.
  • SCoder Performance: SCoder models, fine-tuned on data from bootstrapped synthesizers, achieved state-of-the-art or comparable performance to models trained on much larger amounts of proprietary LLM-distilled data. For example, SCoder-Q14-DS-6.7B surpassed open-source baselines by 5.9% and 9.7% on average on LiveCodeBench and BigCodeBench, respectively.
  • Ablation Studies: Removing any of the key components (multi-checkpoint sampling, multi-aspect scoring, or gradient-based influence estimation) led to a noticeable drop in performance, highlighting the importance of each element in the proposed method.
  • Data Quality: A qualitative analysis using GPT-40 showed that the synthesized data achieved higher scores across various aspects compared to the evol-codealpaca-v1 dataset.

Limitations and Future Work

  • Alternative Synthesis Paradigms: The current framework does not incorporate other promising data generation paradigms like Self-Instruct or Evol-Instruct. Integrating these could further enhance the synthesis process.
  • Domain Specificity: The study is limited to code generation. Applying the methodology to other domains might require tailored approaches and presents challenges in achieving efficient data synthesis.
  • Theoretical Assumptions: While theoretical analysis guarantees convergence under idealized conditions (Lipschitz continuity, contraction property), practical scenarios with non-convex models might not strictly adhere to these. However, empirical results suggest the method is robust.
  • Further Iterations: While performance stabilizes after two to three iterations, exploring the impact of more iterations could be beneficial.

Summary updated

BibTeX
@inproceedings{zhang2025scoder,
  author = {Xinyu Zhang and Changzhi Zhou and Linmei Hu and Luhao Zhang and Xiancai Chen and Haomin Fu and Yang Yang and Mengdi Zhang},
  title = {{SCoder: Progressive Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure