A Shared Low-Rank Adaptation Approach to Personalized RLHF

Renpu Liu, Peng Wang, Donghao Li, Cong Shen, Jing Yang

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively homogeneous and can be captured by a single, unified reward model. This assumption overlooks the inherent diversity and heterogeneity across individuals, limiting the adaptability of RLHF to personalized scenarios and risking misalignments that can diminish user satisfaction and trust in AI systems. In this paper, we address these challenges by introducing Low-Rank Adaptation (LoRA) into the personalized RLHF framework. We apply LoRA in the parameter space of the aggregation of all personalized reward functions, thereby enabling efficient learning of personalized reward models from potentially limited local datasets. Our approach exploits potential shared structures among the local ground-truth reward models while allowing for individual adaptation, without relying on restrictive assumptions about shared representations as in prior works. We further establish sample complexity guarantees for our method. Theoretical analysis demonstrates the effectiveness of the proposed approach in capturing both shared and individual-specific structures within heterogeneous human preferences, addressing the dual challenge of personalization requirements and practical data constraints. Experimental results on real-world datasets corroborate the efficiency of our algorithm in the personalized RLHF setting.

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the standard methodology for aligning Large Language Models (LLMs) with human intent. By training a reward model to mimic human preferences, developers can steer models toward helpfulness and safety. However, a significant limitation of current RLHF implementations is the assumption of a "universal" human preference. In practice, human values are diverse and often conflicting; a summary preferred by a medical professional may differ significantly from one preferred by a student.

Personalizing RLHF presents a triad of challenges: heterogeneity (diverse user values), data scarcity (individual users provide limited feedback), and computational efficiency (training separate models for every user is infeasible). This paper introduces P-ShareLoRA, a framework designed to exploit the shared structural logic of human preferences while specializing in individual nuances.

The P-ShareLoRA Framework

The authors propose Personalized Shared Low-Rank Adaptation (P-ShareLoRA) to bridge the gap between global alignment and individual specialization. Standard LoRA decomposes weight updates into two low-rank matrices, $A$ and $B$. P-ShareLoRA reconfigures this structure to handle multiple users simultaneously:

  • Shared Component ($B$): An "up-projection" matrix shared across all users. This component captures common structural patterns in human evaluation, such as grammatical correctness and general coherence.
  • Personalized Component ($W_i$): A "down-projection" matrix unique to each user $i$. This allows the model to adapt to specific individual preferences, such as preferred length, tone, or level of technical detail.

Initialization Strategies

The effectiveness of P-ShareLoRA was evaluated using three initialization approaches:

  1. Standard (SI): Initializing parameters from scratch.
  2. Global (G): Pre-training the LoRA module on a combined dataset of all users before specializing.
  3. Warm-Up (WU): A hybrid approach involving initial global training steps before diverging into personalized paths.

Theoretical Foundations

Beyond empirical results, the paper provides a rigorous theoretical analysis of the P-ShareLoRA framework. The authors establish an "Individual Expected Value Function Gap," proving mathematically that the learned personalized policy remains near-optimal. Crucially, the analysis demonstrates that this shared structure significantly reduces sample complexity. By leveraging shared parameters, the model requires substantially less data per user to achieve high accuracy compared to training independent models for each individual.

Experimental Results

The researchers benchmarked P-ShareLoRA using the Reddit TL;DR dataset, focusing on the top five most active annotators to simulate distinct user preference profiles. The experiments utilized Llama-3 8B and GPT-J 6B as backbone models.

Performance Metrics

P-ShareLoRA consistently outperformed state-of-the-art (SOTA) baselines in predicting user preferences:

  • Llama-3 8B: Achieved 74.65% accuracy, surpassing the previous SOTA of 73.25%.
  • GPT-J 6B: Achieved 66.93% accuracy, compared to the 66.13% baseline.

Key Insights

  • The Shared B Advantage: The study found that sharing the up-projection matrix ($B$) is more effective than sharing the down-projection matrix ($A$). This suggests that $B$ better preserves common directions for parameter updates in reward modeling.
  • Global vs. Personalized: The "LoRA-Global" baseline—a single model for all users—performed significantly worse for specific individuals. This confirms that "averaging" human values often results in a model that satisfies no one perfectly.
  • Initialization Impact: For complex models like Llama-3, the "Warm-Up" and "Global" initialization strategies proved superior to starting from scratch, highlighting the importance of a strong baseline representation.

Conclusion and Future Work

P-ShareLoRA offers a scalable and theoretically grounded path toward personalized AI. By intelligently partitioning shared and private parameters, the method achieves high alignment accuracy even with limited individual data.

However, several challenges remain for future research. Scaling this approach to millions of users presents significant storage and serving hurdles. Additionally, while this work focuses on reward model accuracy, further study is needed to observe how these models perform in full RLHF generation loops. Finally, as human preferences are dynamic, future iterations of P-ShareLoRA must account for "continual" updates to track evolving user tastes.

Summary updated

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively homogeneous and can be captured by a single, unified reward model. This assumption overlooks the inherent diversity and heterogeneity across individuals, limiting the adaptability of RLHF to personalized scenarios and risking misalignments that can diminish user satisfaction and trust in AI systems. In this paper, we address these challenges by introducing Low-Rank Adaptation (LoRA) into the personalized RLHF framework. We apply LoRA in the parameter space of the aggregation of all personalized reward functions, thereby enabling efficient learning of personalized reward models from potentially limited local datasets. Our approach exploits potential shared structures among the local ground-truth reward models while allowing for individual adaptation, without relying on restrictive assumptions about shared representations as in prior works. We further establish sample complexity guarantees for our method. Theoretical analysis demonstrates the effectiveness of the proposed approach in capturing both shared and individual-specific structures within heterogeneous human preferences, addressing the dual challenge of personalization requirements and practical data constraints. Experimental results on real-world datasets corroborate the efficiency of our algorithm in the personalized RLHF setting.
BibTeX
@inproceedings{liu2025shared,
  author = {Renpu Liu and Peng Wang and Donghao Li and Cong Shen and Jing Yang},
  title = {{A Shared Low-Rank Adaptation Approach to Personalized RLHF}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=lJiq9ykZhl},
}
Figure