LoRe: Personalizing LLMs via Low-Rank Reward Modeling

Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, Maryam Fazel

Abstract

LoRe (Low-Rank Reward Modeling) addresses the limitations of monolithic reward models in Reinforcement Learning from Human Feedback (RLHF) by decomposing human preferences into a shared reward basis and personalized user weights. This low-rank structure allows for efficient personalization and rapid adaptation to new users with minimal feedback, significantly outperforming existing baselines in accuracy and scalability.

Introduction

Standard Reinforcement Learning from Human Feedback (RLHF) typically relies on a "monolithic" reward model—a single mathematical function designed to represent the "average" human preference. However, human values are not uniform; a summary that one person finds perfectly concise might be seen as lacking detail by another. Existing personalization methods often struggle with either rigid categorization (grouping users into broad demographics) or extreme data hunger (requiring extensive feedback from every individual user).

LoRe introduces a middle ground by hypothesizing that while individual preferences are unique, they share underlying dimensions—such as a preference for conciseness, technical depth, or humor. By treating the reward space as low-rank, LoRe can learn these shared dimensions while maintaining individual nuance.

Methodology

The core innovation of LoRe is the decomposition of the reward function into two components:

  1. A Reward Basis ($R_\phi$): A set of $B$ primitive reward functions that serve as the "building blocks" of preference, shared across the entire population.
  2. User Weights ($w_i$): A small, personalized vector for each user that determines their specific weighting of these building blocks.

Mathematically, a user’s personalized reward is expressed as: $p_i = w_i^\top R_\phi$.

Comparison between Monolithic Reward Models and LoRe architecture.

The workflow consists of two stages:

  • Joint Learning: The system learns the shared reward basis and initial user weights from a dataset of "seen" users.
  • Few-Shot Adaptation: For a new user, the reward basis remains frozen. The system only needs to learn a new weight vector ($w_{new}$) from a handful of examples (e.g., 5–10 comparisons), making the process computationally efficient.

Datasets

LoRe was evaluated across three diverse datasets:

  • PersonalLLM: A semi-synthetic dataset simulating diverse user types through responses from eight top-tier LLMs (e.g., GPT-4o, Claude 3).
  • Reddit TLDR: A real-world summarization dataset featuring feedback from 40 human workers.
  • PRISM: A large-scale preference dataset with 1,500 participants from 75 countries, covering over 8,000 live conversations.

Key Results

LoRe consistently outperforms standard models and existing personalization baselines across various benchmarks.

Performance of LoRe vs. baselines on the PersonalLLM dataset across varying levels of preference diversity.

Key findings include:

  • Superior Accuracy: On the PRISM dataset, LoRe achieved 71.0% accuracy in predicting unseen user preferences, compared to 61.0% for the standard Bradley-Terry model and ~59% for other personalization baselines like PAL and VPL.
  • Rapid Adaptation: LoRe exhibits a steep learning curve. Its accuracy improves significantly with just a few additional examples, whereas competing methods often plateau.
  • Efficiency and Scalability: Because it only adds a low-dimensional weight vector per user, LoRe is remarkably parameter-efficient. The authors demonstrate that LoRe requires orders of magnitude fewer trainable parameters than methods like VPL or PAL as the user base scales into the thousands.

Conclusion and Future Work

LoRe demonstrates that human preferences are "compressible"—they can be effectively represented as combinations of a small number of latent factors. By decoupling the features of a good response from the individual's valuation of those features, LoRe provides a scalable path toward truly personalized AI.

While the current model assumes fixed preferences, future work will explore dynamic preferences (where a user's needs change over time) and online RLHF integration, where the model actively explores a user's preferences through conversation.

Summary updated

BibTeX
@inproceedings{bose2025lore,
  author = {Avinandan Bose and Zhihan Xiong and Yuejie Chi and Simon Shaolei Du and Lin Xiao and Maryam Fazel},
  title = {{LoRe: Personalizing LLMs via Low-Rank Reward Modeling}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=bYu4DOqRY8},
}
Figure