Towards Personalized Language Models via Inference-time Human Preference Optimization

Nikki Lijing Kuang, Wei Sun, Scott McFaddin, Yian Ma, Markus EttlAFM @ NeurIPS 2024 · OpenReview

TL;DR

This paper introduces PANDA, a novel method for personalizing Large Language Models (LLMs) at inference time without requiring expensive fine-tuning. PANDA dynamically adapts LLM responses to individual user preferences by leveraging context-aware reward models, offering a flexible and efficient solution for personalized AI interactions.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly powerful, but their current alignment methods primarily focus on optimizing for general human preferences like safety and fairness. This approach falls short when it comes to catering to the diverse and often subjective personal preferences of individual users. Existing methods, which often involve extensive fine-tuning, are computationally prohibitive and lack the adaptability needed for real-time personalization. The paper aims to bridge this gap by enabling LLMs to adapt to individual user preferences dynamically during the generation process, without altering the base model.

Methods

The core of the proposed method, PANDA (Personalized Alignment via Decode-time Adaptation), lies in its decode-time framework. Instead of retraining the LLM, PANDA adjusts the model's output generation during inference. This is achieved by:

  • Context-aware Reward Modeling: For each type of personal preference, a reward model is trained. Crucially, these reward models are designed to be context-aware, meaning they can infer user preferences implicitly from the provided context. This avoids the need for explicit preference inputs from users.
  • Dynamic Adaptation: During the decoding (generation) process, PANDA uses these personalized reward models to dynamically adjust the LLM's token-level predictions. This is done by linearly combining the logits from the base LLM and the reward signal, guided by a context-aware function ($\lambda(x, t)$). This function balances personalization with the LLM's general capabilities, preventing degeneration.
  • Decoupling: PANDA effectively decouples the personalization optimization from the LLM's training process, allowing for flexible and on-the-fly adjustments.

Datasets

The paper evaluates PANDA on two existing benchmark datasets:

  • HH-RLHF (Bai et al., 2022a): This dataset is designed to enhance conversational language models and contains pairwise preference data from human annotators evaluating responses.
  • Personalized Soup (Jang et al., 2023): This dataset is specifically designed to capture diverse human preferences through large-scale pairwise feedback, often collected using simulated human annotations. It generates responses with different preference dimensions like simplicity, expertise, and style.

Key Insights and Ideas

  • Personalization is Key for User Experience: As LLMs move into more personalized applications (assistants, creative tools), catering to individual preferences becomes paramount for user engagement and satisfaction.
  • Decode-time is the Future for Personalization: The computational cost of fine-tuning LLMs makes it impractical for real-time, user-specific personalization. Decode-time methods offer a scalable and efficient alternative.
  • Context is Rich with Preference Signals: Users implicitly encode their preferences in their prompts and interactions. Leveraging this contextual information is a powerful way to infer and adapt to personal tastes without explicit user input.
  • Balancing Act: Effective personalization requires a delicate balance between adhering to personal preferences and maintaining the LLM's general coherence, fluency, and safety. The context-aware $\lambda$ function is crucial for this.

Key Results

PANDA consistently outperformed unaligned LLMs and a prior method (ARGS) across various metrics and tasks, including multi-round chat and single-round instruction following.

  • Improved Personalization: PANDA achieved significantly higher average rewards for personalization across different preference types (e.g., "Comprehensibility," "Specialization"). For instance, in instruction following with the "Specialization" preference, PANDA achieved an average reward of 18.10, compared to 16.76 for ARGS and 1.48 for the unaligned LLM.
  • Maintained Quality: Despite personalization, PANDA maintained semantic coherence and generation diversity, often showing lower repetition rates (rep-2, rep-3, rep-4) than baseline methods.
  • Efficiency: The method is computationally efficient as it avoids retraining or fine-tuning, making it suitable for real-time applications.

Limitations and Future Work

  • Limitations: The paper acknowledges that while PANDA excels at personalization, it relies on pre-trained reward models. The quality of personalization is thus dependent on the quality and coverage of these reward models.
  • Future Work: The authors suggest several promising directions:
  • Developing more effective optimization algorithms for learning preference weights.
  • Exploring more sophisticated reward models that can capture evolving and multi-dimensional user preferences (e.g., for multi-objective tasks).
  • Incorporating multi-modal inputs (images, videos) to further enhance personalization capabilities.

Summary updated

Abstract

The impressive generative capabilities of large language models (LLMs) have led to their widespread adoption across diverse applications. However, existing alignment methods, which rely heavily on expensive fine-tuning processes, focus on optimizing for the \textit{general human preferences} such as safety, fairness, and trustworthiness. These approaches suffer from scalability and adaptability issues when addressing \textit{personal preferences} which could be different across users. In this paper, we introduce a novel approach to LLM alignment for personalized preference based on decode-time frameworks. Our approach enables dynamic adaptation to personal preferences during inference, providing a flexible and computationally efficient solution for personalization without the need of training-time interventions. We demonstrate the efficacy of our method on benchmark datasets and tasks, by enhancing LLMs' ability to adapt to diverse personal preferences compared to the existing alignment methods.
BibTeX
@inproceedings{kuang2024towards,
  author = {Nikki Lijing Kuang and Wei Sun and Scott McFaddin and Yian Ma and Markus Ettl},
  title = {{Towards Personalized Language Models via Inference-time Human Preference Optimization}},
  booktitle = {AFM @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=zXz9BuSD0e},
}
Figure