Abstract
Large language models (LLMs) can acquire strong code-generation capabilities through few-shot learning. In contrast, supervised fine-tuning is still needed for smaller models to achieve good performance. Such fine-tuning demands a large number of task-specific NL-code pairs, which are expensive to obtain. In this paper, we attempt to transfer the code generation ability of an LLM to a smaller model with the aid of weakly-supervised data. More specifically, we propose explicit knowledge transfer (EKT), which uses the few-shot capabilities of a teacher LLM to create NL-code pairs that we then filter for correctness and fine-tune the student on. We evaluate EKT on the task of generating code solutions to math word problems from the GSM8k dataset. We find that EKT not only yields better performance than training with expert iteration, but also outperforms knowledge distillation, another form of knowledge transfer. A GPT-Neo 1.3B model trained using EKT with a GPT-J teacher achieves a 12.4% pass@100 on GSM8k, while the same student and teacher trained with knowledge distillation yield only a 3.7% pass@100. We also show that it is possible for a student model to outperform the teacher using EKT.
Introduction
While massive Large Language Models (LLMs) like GPT-4 exhibit remarkable code-generation capabilities via few-shot prompting, their deployment is often constrained by high computational costs and latency. Smaller models offer a more efficient alternative but typically require extensive supervised fine-tuning to reach competitive performance levels. The primary bottleneck is the scarcity of high-quality natural language (NL) to code pairs.
To address this, researchers have turned to weakly-supervised data—scenarios where the problem and final answer are available, but the intermediate code is missing. This paper introduces Explicit Knowledge Transfer (EKT), a framework designed to bridge this gap by leveraging the few-shot capabilities of a large "teacher" model to synthesize training data for a smaller "student" model.
Methodology: Explicit Knowledge Transfer (EKT)
The EKT framework operates by explicitly filtering the teacher's output to ensure the student learns only from correct reasoning paths. The process consists of four main stages:
- Teacher Generation: A large teacher LLM (e.g., Codex or GPT-J) is prompted with a few examples to generate Python code solutions for a set of math problems.
- Sampling: The teacher generates multiple candidate solutions for each problem.
- Filtering: Each candidate solution is executed. Only programs that produce the correct numerical answer are retained in the "Knowledge Set $\mathbb{K}$."
- Student Fine-tuning: A smaller student model (e.g., GPT-Neo 1.3B) is fine-tuned on this verified Knowledge Set using standard causal language modeling objectives.
Unlike traditional Knowledge Distillation (KD), which requires the student to mimic the teacher's entire probability distribution (including errors), EKT focuses exclusively on successful outcomes. This "explicit" transfer avoids the noise inherent in the teacher's incorrect predictions.
Experimental Setup
The proposed method was evaluated on the GSM8k (Grade School Math 8k) dataset, which contains 8.5k high-quality math word problems. The task is to generate Python code that calculates the correct numerical answer for each problem. Performance is measured using the PASS@k metric, representing the probability that at least one of $k$ generated samples is correct.
Results and Discussion
EKT demonstrates significant improvements over standard baselines. As shown in the results table, EKT consistently outperforms both Knowledge Distillation and Expert Iteration.
Key Findings
- Superiority over Distillation: Using a GPT-J (6B) teacher, EKT achieved a 12.4% PASS@100, nearly 4x the performance of Knowledge Distillation (3.7%).
- Teacher Scaling: The quality of the student is highly dependent on the teacher's "coverage" (the percentage of problems it can solve). When using the more powerful Codex model as a teacher, the GPT-Neo 1.3B student's performance rose to 32.9% PASS@100.
- Specialization: In several instances, the student model outperformed its teacher during greedy decoding. This suggests that fine-tuning on a task-specific knowledge set allows the smaller model to specialize more effectively than a general-purpose teacher using few-shot prompts.
- Architectural Flexibility: Because EKT transfers knowledge through raw text (code) rather than internal model states, it is agnostic to the architectures and tokenizers of the teacher and student models.
Limitations and Future Work
EKT's primary limitation is its reliance on a verifiable signal (e.g., a numerical answer or test cases) to filter the teacher's output. This makes it highly suitable for math and programming tasks but challenging for open-ended creative tasks. Additionally, the student's performance is currently capped by the teacher's initial coverage. Future research could explore iterative EKT, where the student and teacher collaboratively expand the knowledge set to solve increasingly complex problems.
Summary updated