Abstract
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively homogeneous and can be captured by a single, unified reward model. This assumption overlooks the inherent diversity and heterogeneity across individuals, limiting the adaptability of RLHF to personalized scenarios and risking misalignments that can diminish user satisfaction and trust in AI systems. In this paper, we address these challenges by introducing Low-Rank Adaptation (LoRA) into the personalized RLHF framework. We apply LoRA in the aggregated parameter space of all personalized reward functions, thereby enabling efficient learning of personalized reward models from potentially limited local datasets. Our approach exploits potential shared structures among the local ground-truth reward models while allowing for individual adaptation, without relying on restrictive assumptions about shared representations as in prior works. We further establish sample complexity guarantees for our method. Theoretical analysis demonstrates the effectiveness of the proposed approach in capturing both shared and individual-specific structures within heterogeneous human preferences, addressing the dual challenge of personalization requirements and practical data constraints. Experimental results on real-world datasets corroborate the efficiency of our algorithm in the personalized RLHF setting.
Introduction
Standard Reinforcement Learning from Human Feedback (RLHF) has been instrumental in aligning Large Language Models (LLMs) with human expectations. However, the current paradigm typically relies on a "one-size-fits-all" reward model, assuming that human preferences are homogeneous. In practice, what one user considers a high-quality response, another may find verbose or biased. This heterogeneity presents a significant challenge: how can we align models with individual preferences when individual users provide very little feedback data?
Learning a unique reward model for every user from scratch is computationally and data-prohibitive—a problem known as data scarcity. Conversely, a single global model fails to capture the nuances of individual taste. This paper introduces P-ShareLoRA, a framework designed to bridge this gap by leveraging shared structures across users while allowing for efficient individual adaptation.
The P-ShareLoRA Framework
The core innovation of P-ShareLoRA lies in its application of Low-Rank Adaptation (LoRA) to the personalized reward modeling task. While standard LoRA fine-tunes models by adding low-rank matrices to existing weights, P-ShareLoRA structures these updates to exploit commonalities between users.
Shared and Individual Components
In the P-ShareLoRA architecture, the update for a specific user $i$, denoted as $\Delta \Theta_i$, is decomposed into two components:
- Shared Structure ($B$): A shared "basis" matrix that captures common human evaluation patterns across the entire population.
- Individual Adaptation ($W_i$): A user-specific matrix that allows the model to pivot toward an individual's unique preferences.
The total update is formulated as $\Delta \Theta_i = B W_i$. By forcing all users to share the matrix $B$, the model learns general reward patterns from the aggregate dataset, while $W_i$ enables personalization using only the limited feedback provided by a specific individual.
Initialization Strategies
The researchers explored three distinct ways to initialize the shared and individual components:
- Standard Initialization (SI): Training both components from scratch.
- Global (G): Pre-training the shared component on the entire multi-user dataset before introducing individual adaptations.
- Warm-Up (WU): A hybrid approach that uses a limited number of global training steps to stabilize the shared structure before branching into personalized training.
Theoretical Foundations
A key contribution of this work is the establishment of sample complexity guarantees. The authors provide a rigorous proof demonstrating that sharing parameters via the LoRA structure significantly reduces the amount of data required per user to achieve high accuracy. Unlike previous theoretical models that limited personalization to the final linear "head" of the network, P-ShareLoRA allows for low-rank updates throughout the model's parameter space, offering greater expressivity without sacrificing data efficiency.
Experimental Evaluation
To test the framework, the researchers utilized the Reddit TL;DR dataset, a benchmark for text summarization. To simulate a personalized environment, they identified the five most active human labelers in the dataset and treated them as distinct users, with approximately 5,373 training samples per worker. The experiments were conducted using GPT-J 6B and Llama-3 8B as base models.
Key Findings
- Performance Gains: P-ShareLoRA consistently outperformed both "Global" models (trained on all users) and "Local" models (trained only on a single user's data). On Llama-3 8B, the method achieved a prediction accuracy of 74.65%, surpassing the previous state-of-the-art "Share Rep" method (73.25%).
- The Up-Projection Advantage: In traditional LoRA, updates consist of a down-projection ($A$) and an up-projection ($B$). The study found that sharing the up-projection matrix ($B$) yielded superior results compared to sharing the down-projection. This suggests that the latent space of human preferences shares a common structure in how it maps back to the model's high-dimensional weights.
- Data Efficiency: The failure of "Local" baselines confirmed that without shared structure, individual user data is insufficient to train reliable reward models.
Discussion and Limitations
While P-ShareLoRA offers a robust path toward personalized AI, several challenges remain for future research:
- User Scaling: While effective for five users, scaling the management of individual $W_i$ matrices to millions of users presents significant storage and infrastructure hurdles.
- Preference Drift: The current model assumes static preferences, whereas human tastes often evolve over time, necessitating potential "online" or continual learning adaptations.
- Diversity Bias: If a user group holds fundamentally contradictory values, the shared matrix $B$ may struggle to represent the population effectively, potentially introducing bias.
Conclusion
P-ShareLoRA demonstrates that personalization in RLHF is achievable even under strict data constraints. By intelligently sharing low-rank structures across users, AI systems can move beyond generic alignment toward a more nuanced, individual-centric understanding of human values.
Summary updated