TL;DR
This paper introduces Reinforcement Learning with Supervised Alignment (RLA), a novel method that automates LLM alignment by using existing datasets to train a reward model, bypassing costly human labeling and unstable LLM-as-judge approaches. RLA significantly improves LLM generalization, especially on out-of-domain and cross-task evaluations, outperforming SFT and RLAIF in many scenarios.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful but often struggle with generalization beyond their training data, a phenomenon known as overfitting. While methods like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF) aim to align LLMs with human preferences and tasks, they face significant challenges. RLHF is expensive and time-consuming due to human labeling, while RLAIF, which uses LLMs as judges, can suffer from instability, bias, and error propagation. This paper addresses the need for a more efficient, scalable, and stable alignment method that enhances LLM generalization without relying on human annotators or potentially flawed AI judges.
Methods
The core innovation is Reinforcement Learning with Supervised Alignment (RLA), which automates the alignment process. RLA consists of three main steps:
- Supervised Alignment ($\Delta$): This step constructs a preference-based dataset. For each query in an existing linguistic dataset, the target LLM generates multiple responses. The semantic similarity between each response and the ground truth label is computed. Responses with higher similarity are considered "chosen," and those with lower similarity are "rejected." These "chosen" and "rejected" pairs form the supervised alignment dataset. This approach leverages cosine similarity as a proxy for response quality.
- Reward Model ($\Sigma_{\Delta}$): A reward model is trained on the supervised alignment dataset ($\Delta$). This model learns to predict how likely a candidate response is the correct output for a given query. This is framed as a supervised fine-tuning (SFT) task, using a Bradley-Terry ranking loss.
- Reinforcement Learning Framework (RLA): The target LLM (policy network $\pi_{\phi}$) is then optimized using Proximal Policy Optimization (PPO) and the trained reward model ($\Sigma_{\Delta}$). This step aims to improve the LLM's ability to generate responses that are favored by the reward model, thereby enhancing generalization.
Datasets
- MS-MARCO: Used for training the reward model and for in-domain QA evaluation. This dataset contains mixed short open-ended and span label formats, with web context snippets.
- TriviaQA: Used for out-of-domain QA evaluation. This dataset consists of multiple subsets and primarily provides URLs and text snippets, with shorter average label lengths and no direct web context for each query.
- TruthfulQA (MCQA): Used for cross-task few-shot evaluation, specifically Multiple-Choice QA (MCQA). This benchmark is designed to assess performance on adversarial questions and does not provide web-based context.
- UniLC: Used for cross-task zero-shot evaluation, specifically Fact and Fairness Verification (FFV). This benchmark includes subsets from Climate-FEVER, Health-PUB, HSD, and SBIC, requiring binary classification.
Key Insights and Ideas
- Cosine Similarity as a Reward Proxy: The paper posits that semantic similarity between a generated response and the ground truth label is a strong indicator of response quality. This insight allows for the creation of a preference dataset without human judgment.
- Automated Alignment: RLA successfully automates the alignment process, removing the need for expensive human annotation or the potential pitfalls of LLM-as-judge systems. This makes LLM alignment more scalable and cost-effective.
- Improved Generalization: The core finding is that RLA significantly enhances LLM generalization, particularly on tasks and data distributions that differ from the training set. This is a crucial step towards more robust and reliable LLMs.
- Stability and Efficiency: By using existing datasets and a supervised approach for reward modeling, RLA offers greater stability and is more efficient than traditional RLHF, requiring less data and computational resources.
Key Results
- Outperforms RLAIF: RLA achieved relative improvements over RLAIF ranging from +5.38% to +131.8% on the MS-MARCO dataset.
- Enhances Baseline LLM: RLA significantly improved the baseline Llama3 LLM, achieving up to +55% improvement on in-domain tasks and up to +16% on out-of-domain tasks.
- Surpasses SFT on Out-of-Domain Tasks: While RLA slightly underperformed supervised fine-tuning (SFT) on in-domain benchmarks, it surpassed SFT by up to 50x on out-of-domain and cross-task evaluations, demonstrating superior generalization capabilities.
- Effective on Diverse Tasks: RLA showed strong performance across various tasks, including QA (in-domain and out-of-domain), MCQA, and Fact and Fairness Verification, often outperforming SFT, especially when web context was not provided.
Limitations and Future Work
- Inherited Biases: RLA may inherit biases present in the MS-MARCO dataset, particularly in underrepresented areas like complex fact-checking.
- Reliance on Dataset Labels: The method relies on dataset labels being treated as ground truth, which may not always reflect factual correctness.
- Web Search Amplification: Reliance on web search can amplify misleading snippets.
- Future Work: The authors suggest refining retrieval mechanisms, expanding training datasets for scalability, and improving fine-tuning strategies. They also note that while their method is cost-effective, further research into optimizing runtime could yield even better results.
Summary updated