Abstract
Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for general purposes. Thus, the demands for customizing publicly available LLMs emerge, but are currently under-studied. In this work, we consider customizing pre-trained LLMs with new human preferences. Specifically, the LLM should not only meet the new preference but also preserve its original capabilities after customization. Drawing inspiration from the observation that human preference can be expressed as a reward model, we propose to cast LLM customization as optimizing the sum of two reward functions, one of which (denoted as $r_1$) was used to pre-train the LLM while the other (denoted as $r_2$) characterizes the new human preference. The obstacle here is that both reward functions are unknown, making the application of modern reinforcement learning methods infeasible. Thanks to the residual Q-learning framework, we can restore the customized LLM with the pre-trained LLM and the residual Q-function without the reward function $r_1$. Moreover, we find that for a fixed pre-trained LLM, the reward function $r_2$ can be derived from the residual Q-function, enabling us to directly learn the residual Q-function from the new human preference data upon the Bradley-Terry model. We name our method Q-Adapter as it introduces an adapter module to approximate the residual Q-function for customizing the pre-trained LLM towards the new preference. Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of Q-Adapter on both retaining existing knowledge and learning new preferences.
Introduction
Open-source Large Language Models (LLMs) like Llama 3 are powerful generalists. However, real-world applications often require specialization—adhering to specific brand voices, domain-specific knowledge (e.g., legal or medical), or stricter safety guidelines. The standard approach to customization is fine-tuning via Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF).
Unfortunately, these methods often suffer from catastrophic forgetting: as the model learns new preferences, it loses its proficiency in reasoning, mathematics, or general knowledge. While KL-divergence constraints are commonly used to mitigate this, they are often insufficient to prevent performance degradation. The fundamental challenge lies in the lack of access to the original reward function ($r_1$) used during pre-training, making it difficult to optimize for both old and new objectives simultaneously.
Methodology: The Q-Adapter Framework
The authors propose Q-Adapter, a framework that treats LLM customization as a reward maximization problem where the objective is the sum of the original reward ($r_1$) and the new preference reward ($r_2$).
Residual Q-Learning
To overcome the unknown nature of $r_1$, Q-Adapter utilizes a residual Q-learning framework. The optimal customized policy is expressed as a combination of the frozen pre-trained model and a "residual Q-function." This function represents the gap between the original model's values and the new desired preferences. Crucially, the pre-trained model itself acts as a proxy for $r_1$, allowing $r_1$ to be mathematically canceled out of the optimization objective.
Direct Learning from Preferences
Instead of the traditional multi-stage RLHF pipeline (reward model training followed by PPO), Q-Adapter learns the residual Q-function directly from preference pairs using the Bradley-Terry model. This reduces optimization errors and computational overhead.
Adapter Architecture
The residual Q-function is implemented as a lightweight LoRA (Low-Rank Adaptation) module. During inference, the final response is generated by combining the log-probabilities of the frozen base LLM with the outputs of the Q-Adapter. This architecture ensures that the base model's knowledge is preserved while the adapter steers the output toward the new preference.
Experimental Evaluation
The researchers evaluated Q-Adapter across several scenarios:
- Domain-Specific Preference (DSP): 52K instructions across Academy, Business, Entertainment, and Literature domains to test stylistic adaptation.
- HH-RLHF (Helpful and Harmless): A sequential task taking a "helpful" model and customizing it for "harmlessness."
- General Benchmarks: MMLU (general knowledge), GSM8k (math), BBH (reasoning), and IFEval (instruction following) were used to measure forgetting.
Key Results
- Mitigating Forgetting: On the MMLU benchmark, Q-Adapter maintained stable performance close to the base model's original scores, whereas standard DPO and Replay methods showed significant declines.
- Better Customization: GPT-4o evaluations showed that Q-Adapter achieved higher win rates against SFT baselines compared to DPO or Replay-based methods in all DSP domains.
- Balanced Alignment: In HH-RLHF tests, Q-Adapter successfully enhanced harmlessness without causing "over-refusal" on benign prompts.
Conclusion and Future Work
Q-Adapter provides a robust solution for customizing LLMs while mitigating catastrophic forgetting. By bypassing the need for an explicit reward model and leveraging the base model's implicit knowledge, it offers a stable and efficient path for specialization. Future work includes exploring adaptations for models trained only via SFT and investigating multi-round, continuous customization with multiple adapters.
Code is available at https://github.com/mansicer/Q-Adapter.
Summary updated