Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An

TL;DR

Q-Adapter is a novel framework for customizing pre-trained Large Language Models (LLMs) to new human preferences without suffering from "catastrophic forgetting." By framing customization as a multi-reward optimization problem and leveraging residual Q-learning, the authors enable models to learn new specialized traits while mathematically preserving the general knowledge and reasoning capabilities of the original base model.

What Problem Does This Paper Address?

Open-source LLMs like Llama 3 are powerful generalists, but real-world applications often require them to be specialists—aligned with specific domain knowledge (e.g., medical or legal) or personalized user preferences. The standard approach to this is fine-tuning via Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF).

However, these methods suffer from a significant "alignment tax": as the model learns new preferences, it often forgets the general capabilities it acquired during pre-training. This is known as catastrophic forgetting. A model might become an expert in "harmlessness" but lose its ability to solve complex math problems or follow intricate instructions. The challenge is to maximize a new reward (the new preference) while simultaneously maximizing the original, unknown reward that the base model was initially trained to satisfy.

Figure 1: An example of adapting a pre-trained LLM while preserving its original knowledge. Left: Suppose that we have a pre-trained helpful LLM. After adapting it to preference data of harmlessness, we would like it to be both helpful and harmless. Right: A corresponding case. The customized LLM not only inherits helpful knowledge from the pre-trained LLM, but also be much more harmless by learning from preference data on harmlessness.

Methods

The authors propose Q-Adapter, which treats LLM customization as optimizing the sum of two reward functions: $r_1$ (the original general preference) and $r_2$ (the new specific preference).

The technical hurdle is that both $r_1$ and $r_2$ are typically unknown. To bypass this, the authors utilize the Residual Q-learning (RQL) framework. The key insight is that the pre-trained LLM already implicitly contains information about $r_1$. By defining a "residual Q-function" ($\hat{Q}$), they can represent the difference between the original optimal policy and the new customized optimal policy.

Key steps include: 1. Direct Learning from Data: Using the Bradley-Terry model (a standard for modeling pairwise preferences), the authors derive a loss function that allows them to learn the residual Q-function directly from preference data without ever needing to explicitly reconstruct the reward functions. 2. Adapter Architecture: The residual Q-function is approximated using a lightweight adapter module (implemented via LoRA). 3. Inference-Time Fusion: During generation, the final policy is derived by combining the log-probabilities of the frozen base model with the output of the adapter. This ensures the base model's "knowledge" remains physically intact and is simply steered by the adapter.

Datasets

The researchers evaluated Q-Adapter across two primary scenarios:

  • Domain-Specific Preference (DSP): A dataset categorized into four domains: Academy, Business, Entertainment, and Literature. This tests the model's ability to adopt specific styles.
  • HH-RLHF (Helpful and Harmless): A standard benchmark for alignment. The authors specifically tested "customization between tasks"—taking a model already aligned for helpfulness and adapting it for harmlessness to see if it could maintain both traits.

Key Insights and Ideas

  • Reward Interchangeability: The paper provides a mathematical proof (Corollary 4.1) showing a one-to-one correspondence between feasible reward functions and residual Q-functions. This justifies why learning the Q-function directly is just as effective as (and more efficient than) learning a reward model first.
  • Beyond KL-Regularization: Traditional methods like DPO use KL-divergence to keep the new model "close" to the old one. Q-Adapter goes further by explicitly aiming to maximize the original reward $r_1$. The authors found that while KL-divergence helps, it still allows for significant forgetting as training progresses, whereas Q-Adapter remains stable.
  • Seamless Integration: Because the base model is frozen, Q-Adapter is highly efficient. It avoids the computational cost of full parameter fine-tuning and allows for "plug-and-play" customization where different adapters could theoretically be swapped for different tasks.

Key Results

Q-Adapter consistently outperformed baselines like SFT, DPO, PPO, and Replay-based methods in both preference alignment and knowledge retention.

  • Preserving General Capabilities: In the DSP tasks, Q-Adapter maintained significantly higher scores on general benchmarks like MMLU and GSM8k. For example, in the "Business" domain, Q-Adapter achieved an MMLU score of 66.81, nearly matching the base model (67.68), while DPO dropped to 59.84.
Table 1: Evaluation on r1 in DSP: general benchmark scores of Q-Adapter, SFT, Replay, PR (DPO), and PR (PPO) fine-tuned in data from four specific domains in DSP. The scores represent accuracy in MMLU, MMLU Pro, flexible extraction accuracy in GSM8k, and average accuracy in IFEval over each level and matching.
  • Superior Alignment: When judged by GPT-4o, Q-Adapter achieved higher win rates against SFT than other methods across all four DSP domains, particularly excelling in the "Entertainment" category with a 63.77% win rate.
Table 2: Evaluation on r2 in DSP: win rates of Q-Adapter, Replay, PR (DPO), and PR (PPO) fine- tuned in data from four specific domains in DSP against the SFT baseline.
  • Qualitative Success: In the HH-RLHF experiments, Q-Adapter was the only method that successfully provided a helpful response (e.g., a pumpkin pie recipe) while maintaining harmlessness, whereas baselines often became "over-aligned" and refused to answer harmless prompts.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Base Model Assumptions: The current derivation assumes the base model was trained using RLHF. Investigating how to adapt models trained solely via SFT is a future goal.
  • Single Round of Customization: The experiments focused on one round of adaptation. While the framework supports continuous, multi-stage customization, this hasn't been empirically tested yet.
  • Data Sensitivity: Like all preference-based methods, the quality, bias, and noise in the training data can still impact the final model's performance.
  • Multi-Agent Systems: The authors suggest that using multi-agent RL to customize groups of LLMs is a promising future direction.

Summary updated

Abstract

Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for \emph{general} purposes. Thus, the demands for customizing publicly available LLMs emerge, but are currently under-studied. In this work, we consider customizing pre-trained LLMs with new human preferences. Specifically, the LLM should not only meet the new preference but also preserve its original capabilities after customization. Drawing inspiration from the observation that human preference can be expressed as a reward model, we propose to cast LLM customization as optimizing the sum of two reward functions, one of which (denoted as $r_1$) was used to pre-train the LLM while the other (denoted as $r_2$) characterizes the new human preference. The obstacle here is that both reward functions are unknown, making the application of modern reinforcement learning methods infeasible. Thanks to the residual Q-learning framework, we can restore the customized LLM with the pre-trained LLM and the \emph{residual Q-function} without the reward function $r_1$. Moreover, we find that for a fixed pre-trained LLM, the reward function $r_2$ can be derived from the residual Q-function, enabling us to directly learn the residual Q-function from the new human preference data upon the Bradley-Terry model. We name our method Q-Adapter as it introduces an adapter module to approximate the residual Q-function for customizing the pre-trained LLM towards the new preference. Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of Q-Adapter on both retaining existing knowledge and learning new preferences. Code is available at https://github.com/mansicer/Q-Adapter.
BibTeX
@inproceedings{li2025q,
  author = {Yi-Chen Li and Fuxiang Zhang and Wenjie Qiu and Lei Yuan and Chengxing Jia and Zongzhang Zhang and Yang Yu and Bo An},
  title = {{Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=WLSrq1254E},
}
Figure