Abstract: As the diversity of users increases, the capability of providing personalized responses by large language models (LLMs) has become increasingly important. Existing approaches have only limited successes in LLM personalization, due to the absence of personalized learning or the reliance on shared personal data. This paper proposes a new approach for a few-shot personalization of LLMs with their mis-aligned responses (FERMI). Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs, based on user profile (e.g., demographic information) and a few examples of previous opinions. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs, which are especially crucial for the effective personalization of LLMs. In addition, we develop an effective inference method to further leverage the context of the test query and the personalized prompts. Our experimental results demonstrate that FERMI significantly improves performance across various benchmarks, compared to best-performing baselines.
Introduction
Large Language Models (LLMs) often struggle with a "one-size-fits-all" approach, failing to account for the diverse backgrounds, ideologies, and preferences of individual users. While prompt engineering and learning-based methods exist, they are either manually intensive or raise significant privacy concerns by requiring shared user data.
FERMI (Few-shot Personalization of LLMs with Mis-aligned Responses) addresses these challenges by steering models using only a tiny amount of user data—such as a short profile and 20 previous opinions—without retraining or data sharing. The core innovation lies in analyzing the model's specific "mis-aligned responses" (errors) to iteratively refine personalized instructions.
The FERMI Framework
FERMI treats prompt optimization as an iterative search process managed by an "optimizer" LLM. Unlike standard optimizers that rely solely on numerical scores, FERMI leverages the qualitative context of failures.
The process consists of four main components:
- Iterative Optimization: Starting with a basic prompt, FERMI tests it against a user's few-shot history.
- Learning from Mis-alignment: FERMI provides the optimizer with the context of mis-aligned responses—specific questions the current prompt failed on and the model's incorrect reasoning. This allows the optimizer to "debug" and refine the prompt effectively.
- Memory Management: An optimization memory maintains
<prompt, score, context>triplets, enabling the generation of new prompts that build on previous lessons. - Retrieval-of-Prompt (RoP): During inference, FERMI uses a retrieval mechanism to select the personalized prompt that performed best on historical examples most similar to the current test query.
Incorporating Mis-aligned Context
The "secret sauce" of FERMI is how it updates its optimization memory. By explicitly adding the contexts of mis-aligned responses, the system provides the optimizer LLM with the necessary information to identify patterns in its own bias and adjust the prompt accordingly.
Experimental Evaluation
The researchers evaluated FERMI across several benchmarks, including OpinionQA (US demographics), GlobalOpinionQA (cross-national perspectives), and the LaMP benchmark (tagging, rating, and titling tasks).
Key Results
- Performance Gains: FERMI achieved an average accuracy improvement of 6.8% on OpinionQA and 4.1% on GlobalOpinionQA compared to top baselines.
- Superiority over OPRO: FERMI outperformed OPRO (a standard LLM-based prompt optimizer) by 4.05% on average.
- Instruction Efficiency: Converting user opinions into FERMI's detailed instruction format improved performance from 56.3% to 66.4%, even without changing the underlying data.
Insights and Discussion
FERMI's success stems from several key factors:
- Errors as Teachers: Providing specific examples of mis-aligned responses is more effective than numerical scores alone, as it helps the model identify and correct its own biases.
- Persona Distillation: Optimized prompts often evolve into detailed "distilled personas"—narrative guidelines that are easier for models to follow than raw Q&A pairs.
- Cross-Model Transferability: Prompts optimized on strong models like GPT-4 can be successfully transferred to smaller models like LLaMA-2, providing a significant performance boost.
Limitations
While effective, the optimization phase can be computationally expensive due to multiple high-end LLM calls. Additionally, while FERMI avoids cross-user data sharing, it still involves sending personal data to LLM APIs. Future work may focus on real-time "continual optimization" and implementing filters to prevent the generation of socially negative prompts.
Summary updated