Abstract
As the diversity of users increases, the capability of providing personalized responses by large language models (LLMs) has become increasingly important. Existing approaches have only limited successes in LLM personalization, due to the absence of personalized learning or the reliance on shared personal data. This paper proposes a new approach for a few-shot personalization of LLMs with their mis-aligned responses (FERMI). Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs, based on user profile (e.g., demographic information) and a few examples of previous opinions. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs, which are especially crucial for the effective personalization of LLMs. In addition, we develop an effective inference method to further leverage the context of the test query and the personalized prompts. Our experimental results demonstrate that FERMI significantly improves performance across various benchmarks, compared to the best-performing baselines.
Introduction
Large Language Models (LLMs) typically function as "one-size-fits-all" systems, reflecting the majority biases present in their massive training datasets. This lack of personalization becomes evident when users pose subjective questions or require responses tailored to specific demographic, cultural, or personal backgrounds. Standard models often fail to capture these individual nuances.
Existing personalization methods generally fall into two categories: manual prompt engineering and fine-tuning. Manual engineering is unscalable and expensive, while fine-tuning often requires large amounts of shared user data, raising significant privacy concerns and requiring substantial computational resources. FERMI (Few-shot Personalization of LLMs with Mis-aligned Responses) addresses these challenges by steering LLMs toward individual preferences using only a "few-shot" sample of previous opinions without retraining the model.
Methodology: The FERMI Framework
FERMI treats personalization as a prompt optimization problem, utilizing a high-capacity "optimizer" LLM (e.g., GPT-4) to iteratively refine instructions for a "prediction" LLM. The framework consists of two main components: an iterative optimization loop and a specialized inference method called Retrieval-of-Prompt (RoP).
Iterative Prompt Optimization
The core of FERMI is its ability to learn from its own mistakes. The optimization process follows three steps:
- Score Prompts: Candidate prompts are evaluated against a small set of known user opinions. FERMI identifies "mis-aligned responses"—instances where the model's prediction contradicts the user's actual stated opinion.
- Update Memory: The system maintains an "Optimization Memory" containing triplets of
<prompt, score, context>. Unlike standard optimizers that rely solely on numerical scores, FERMI includes the specific failure cases (the mis-aligned responses) as context. This provides a rich "critique" for the optimizer. - Generate New Prompts: The optimizer LLM analyzes the memory to generate refined prompts that attempt to correct the observed error patterns.
Retrieval-of-Prompt (RoP) Inference
To further enhance performance during inference, FERMI employs Retrieval-of-Prompt. Rather than relying on a single global prompt, RoP uses embeddings to identify which previous user opinions are most relevant to a new test query. It then selects the personalized prompt that performed best on those specific types of questions, ensuring the most contextually appropriate persona is applied.
Experimental Evaluation
FERMI was evaluated across several diverse benchmarks to test its robustness:
- OpinionQA: Views from 60 US demographic groups on social topics.
- GlobalOpinionQA: Cross-national survey data from 46 countries.
- LaMP Benchmark: Tasks including movie tagging, review rating, and paper title generation.
Key Results
- Performance Gains: FERMI achieved a 6.8% average accuracy improvement on OpinionQA and a 4.1% improvement on GlobalOpinionQA over state-of-the-art baselines.
- Superior Optimization: Compared to OPRO (a leading prompt optimization method), FERMI showed a 4.05% improvement in average accuracy, demonstrating the value of including mis-aligned response contexts.
- Sample Efficiency: FERMI is highly efficient, achieving higher accuracy with only 10 user opinions than OPRO could achieve with 20.
- Cross-Model Transferability: Prompts optimized on one model (e.g., ChatGPT) remained highly effective when transferred to others (e.g., LLaMA-2-70B), suggesting that FERMI captures a model-agnostic representation of user personas.
Discussion and Conclusion
A central insight of this work is that errors are valuable signals. By providing the optimizer LLM with specific examples of how it failed, FERMI enables the model to "reason" about a user's unique perspective, often resulting in sophisticated natural language personas. While the computational cost of using high-end LLMs for optimization is a factor, the emergence of more efficient models like GPT-4o-mini makes this approach increasingly viable. FERMI represents a significant step toward effective, privacy-preserving, and sample-efficient personalization for large language models.
Summary updated