TL;DR
This paper introduces Distilled Self-Critique (dSC), a novel method for aligning Large Language Models (LLMs) by framing Reinforcement Learning from AI Feedback (RLAIF) as Bayesian inference. dSC uses a Gibbs sampler to refine LLM outputs based on synthetic data and then distills this process into a fine-tuned model, offering a cost-effective alternative to existing alignment techniques.
What Problem Does This Paper Address?
Aligning LLMs with human values, ensuring they are safe, ethical, and helpful, is a critical challenge. Traditional methods like Reinforcement Learning from Human Feedback (RLHF) and its AI-driven counterpart, RLAIF, rely on reward models to guide LLM behavior. However, these methods can be complex and data-intensive. The paper aims to provide a more efficient and interpretable approach to LLM alignment by leveraging a Bayesian perspective and generating synthetic data for training, thereby reducing reliance on expensive human feedback.
Methods
The core of the proposed method, Distilled Self-Critique (dSC), is built upon a Bayesian interpretation of RLAIF. The process involves the following key steps:
- Critique Step: Given an LLM's generated response ($x$) for a prompt ($c$), the LLM is prompted to generate a critique of its own output. This critique aims to identify undesirable aspects, such as harmful content. This step is performed independently of a reward model.
- Revision Step: Using the generated critique, the LLM then revises its original response. This revision process is framed as sampling from a posterior distribution that incorporates a reward model ($r(x)$) as a likelihood term. Specifically, the probability of a revised response $x'$ is proportional to $p(x'|x_c, c) \exp(r(x'))$, where $p(x'|x_c, c)$ is the LLM's generation probability given the critique and context, and $\exp(r(x'))$ acts as a likelihood that favors high-reward outputs. A Metropolis acceptance step is used to decide whether to accept the revised response. This critique-revision cycle is iterated, forming a Gibbs sampler chain.
- Distillation Step: Running the Gibbs sampler can be computationally expensive. To address this, dSC employs a self-distillation phase. The accepted synthetic samples generated by the MCMC chain are used to fine-tune the original LLM. This is achieved by minimizing the reverse KL divergence between the original LLM's distribution and the distribution of the accepted samples, which is akin to behavioral cloning. The authors use a LoRA adapter for efficient fine-tuning.
This approach differs from methods like Self-Refine by incorporating a likelihood model to un-bias samples and a self-distillation phase. It also distinguishes itself from ReST by explicitly including critique and revision steps.
Datasets
The paper does not introduce new benchmark datasets. Instead, it utilizes existing datasets and generates synthetic data for training and evaluation. Specifically:
- Harmful Behaviors: Adversarial prompts from the Harmful Behaviors dataset (Zou et al., 2023) are used.
- Sentiment and Privacy: Synthetic datasets are generated for experiments on avoiding negative sentiment and preserving privacy. The code for generating these synthetic datasets is made publicly available.
Key Insights and Ideas
- Bayesian Interpretation of RLAIF: Framing RLAIF as Bayesian inference provides a principled way to understand and implement alignment. The reward model naturally fits into this framework as a likelihood function.
- Synthetic Data for Alignment: Generating synthetic data through a critique-revision process allows for LLM alignment without direct human feedback, making the process potentially cheaper and more scalable.
- Gibbs Sampling for Refinement: Using a Gibbs sampler for critique and revision iteratively refines LLM outputs, allowing for a more thorough exploration of the response space towards desired attributes.
- Self-Distillation for Efficiency: Distilling the MCMC refinement process into a fine-tuned model makes the alignment practical for deployment, as it avoids the need for iterative sampling at inference time.
- Viable and Cheap Alternative: The experimental results suggest that dSC can be a competitive and cost-effective alternative to existing alignment methods.
Key Results
The dSC method demonstrated strong performance across several tasks:
- Avoiding Harmful Behaviors: Distilled Self-Critique achieved the highest safety scores for both evaluated model sizes (Stable Beluga 7B and 13B), reaching 0.90 and 0.92 respectively, outperforming baselines like prompting and Self-Critique (SC).
- Avoiding Negative Sentiment: The dSC method, when applied to Stable-Beluga 7B, achieved a median sentiment score of 5.0, matching the performance of the Self-Critique (SC) baseline and significantly improving over the original model (1.0).
- Privacy-Preserving Generation: For this task, dSC applied to Mistral 7B achieved a significantly lower NER score (0.35) compared to the original model (4.40) and the SC baseline (1.00), indicating a better ability to avoid mentioning personal identifiable information.
- Transfer Learning: The paper shows that synthetic data generated with Stable-Beluga-7B using dSC can be used to fine-tune other smaller LLMs (e.g., Phi-1.5B), indicating the generalizability of the generated alignment data.
Limitations and Future Work
- Computational Cost of Sampling: While distillation mitigates this for inference, the initial Gibbs sampling process can still be computationally intensive.
- Choice of Divergence: The paper mentions that future work could explore alternative divergences, such as contrastive divergence, instead of the reverse KL divergence used for distillation.
- Zero-Shot Reward Models: The authors suggest investigating the use of zero-shot reward models for the filtering process, which could further simplify the alignment pipeline.
- Scalability: While promising, the scalability of this approach to extremely large LLMs and complex alignment objectives warrants further investigation.
Summary updated