Applying RLAIF for Code Generation with API-usage in Lightweight LLMs

Sujan Dutta, Sayantan Mahinder, Raviteja Anantha, Bortik BandyopadhyayACL 2024

TL;DR

This paper introduces a Reinforcement Learning from AI Feedback (RLAIF) framework to improve code generation capabilities in lightweight LLMs, specifically focusing on accurate API usage. By leveraging a larger LLM to generate feedback, they train a reward model that significantly enhances smaller models, even outperforming larger baselines.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown remarkable abilities in various natural language tasks. Reinforcement Learning from Human Feedback (RLHF) has been a key technique for aligning LLM outputs with human preferences. However, RLHF is expensive due to the need for high-quality human feedback. Reinforcement Learning from AI Feedback (RLAIF) offers a scalable alternative by using AI to generate feedback.

A specific challenge in LLM code generation is the accurate use of APIs, often hindered by the "hallucination" problem. While many LLMs can generate code, ensuring they correctly call external tools (APIs) is difficult. Existing methods for code generation often rely on executing the generated code to provide feedback, which is computationally expensive and not always feasible for complex API interactions. Furthermore, research on improving code generation with RLAIF, especially for lightweight LLMs (under 1 billion parameters), is limited. This paper aims to bridge this gap by developing an RLAIF framework for lightweight LLMs to generate code with correct API usage.

Methods

The proposed framework follows a three-step pipeline:

  1. Training a Base Model: A lightweight LLM (GPT-2-large, 780M parameters) is fine-tuned on the Gorilla dataset using supervised learning. This creates a base model, denoted as $M_{SF T}$.
  2. Training a Reward Model using LLM Feedback: Instead of human annotators, a larger LLM (GPT-3.5) is used to provide feedback. A specialized prompting strategy is employed, where GPT-3.5 is asked a series of binary (yes/no) questions about the quality of generated code and its API usage. These questions cover aspects like functionality, correct imports, API usage, bug-freeness, and syntax. The number of "yes" answers from GPT-3.5 is used to score the generated code. This scored data is then used to train a reward model ($M_{reward}$) that learns to classify whether a machine-generated code is acceptable for a given instruction.
  3. Reinforcement Learning: The base model ($M_{SF T}$) is further fine-tuned using the Proximal Policy Optimization (PPO) algorithm. The reward signal for this RL step is derived from the logit scores of the trained reward model ($M_{reward}$). The final fine-tuned model is denoted as $M_{RL}$.

Datasets

The paper uses the Gorilla dataset for training and evaluation. Specifically, they focus on the HuggingFace part of the dataset, which contains over 925 unique APIs across 37 domains. Each instance includes an instruction, domain, API call, explanation, and a complete Python script. The dataset is particularly relevant because the Python scripts often involve downloading and loading ML models from HuggingFace, requiring internet connectivity and significant computational resources, making traditional execution-based feedback methods difficult.

Key Insights and Ideas

  • AI Feedback for API Usage: The core idea is that AI feedback, generated through carefully crafted prompts, can effectively guide LLMs to improve their API usage in code generation, a notoriously difficult task.
  • Scalability of RLAIF: By replacing human annotators with a larger LLM for feedback generation, the RLAIF approach significantly reduces the cost and time associated with fine-tuning LLMs for complex tasks like code generation.
  • Lightweight Models Can Compete: The research demonstrates that even smaller LLMs, when fine-tuned with RLAIF, can achieve performance comparable to, or even surpassing, much larger models. This has significant implications for deploying capable models on resource-constrained devices.
  • Multi-faceted Quality Assessment: Using multiple binary questions to assess code quality (functionality, imports, API correctness, etc.) provides a more nuanced feedback signal than a single score, leading to better reward model training.

Key Results

  • The proposed RLAIF framework significantly enhances the performance of the fine-tuned LLM baseline ($M_{SF T}$).
  • The final RLAIF-tuned model ($M_{RL}$) achieves a 4.5% improvement in executability rate compared to the baseline.
  • Remarkably, the lightweight $M_{RL}$ model (780M parameters) outperforms a much larger fine-tuned baseline (MGorilla, 7B parameters, which is fine-tuned LLAMA-7B) by 1.0% in code executability rate, despite having nine times fewer parameters.
  • $M_{RL}$ also shows improvements in CodeBLEU (1.6 points) and AST match (0.66%) over the MGorilla baseline.
  • Figure 2 illustrates a qualitative improvement where $M_{RL}$ correctly defines a variable that was left undefined in both MGorilla and $M_{SF T}$, leading to an error.

Limitations and Future Work

  • Bias: Like most LLMs, the models inherit biases from their pre-trained base model (GPT-2-large), which could lead to biased code comments.
  • Language Diversity: The current work is limited to Python code generation. Future work should expand to other programming languages like C++, Java, and JavaScript.
  • API Frequency: The performance difference between frequent (head) and infrequent (tail) APIs was not analyzed, suggesting potential for improvement by focusing on tail APIs.
  • Offline Learning: The learning methodology is offline. Given the rapid evolution of ML models and APIs, periodic updates to the model are necessary. The RLAIF framework's reduced resource intensity makes these updates more feasible.

Summary updated

BibTeX
@inproceedings{dutta2024applying,
  author = {Sujan Dutta and Sayantan Mahinder and Raviteja Anantha and Bortik Bandyopadhyay},
  title = {{Applying RLAIF for Code Generation with API-usage in Lightweight LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure