Abstract
As large language models (LLMs) demonstrate increasingly advanced capabilities, aligning their behaviors with human values and preferences becomes crucial for their wide adoption. While previous research focuses on general alignment to principles such as helpfulness, harmlessness, and honesty, the need to account for individual and diverse preferences has been largely overlooked, potentially undermining customized human experiences. To address this gap, we train LLMs that can “interact to align”, essentially cultivating the meta-skill of LLMs to implicitly infer the unspoken personalized preferences of the current user through multi-turn conversations, and then dynamically align their following behaviors and responses to these inferred preferences. Our approach involves establishing a diverse pool of 3,310 distinct user personas by initially creating seed examples, which are then expanded through iterative self-generation and filtering. Guided by distinct user personas, we leverage multi-LLM collaboration to develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. Finally, we apply supervised fine-tuning and reinforcement learning to enhance LLMs using this dataset. For evaluation, we establish the ALOE (ALign with custOmized prEferences) benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations. Experimental results demonstrate the effectiveness of our method in enabling dynamic, personalized alignment via interaction. The code and dataset will be made public.
Introduction
Most Large Language Models (LLMs) are currently aligned to a "one-size-fits-all" standard of being helpful, harmless, and honest. While this ensures safety, it often results in a rigid conversational style that ignores individual diversity. This paper introduces a framework to train LLMs to "interact to align," allowing them to implicitly infer a user's unique, unspoken preferences through multi-turn conversation and dynamically tailor their responses. By leveraging a new tree-structured preference dataset and the ALOE benchmark, the authors demonstrate a 32% relative improvement in personalized alignment over mainstream models.
The Challenge of Individual Alignment
Current alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), focus on general human values. However, an extroverted artist in Los Angeles has different expectations for a conversation than a reserved engineer in Tokyo. Users rarely state these preferences explicitly. To be truly helpful, a model must possess the "meta-skill" of inference—picking up on cues like tone, location, family status, or hobbies over several turns and adjusting its behavior accordingly. Existing models struggle with this dynamic adaptation, often reverting to generic personas regardless of the user's input.
Methodology: Interact to Align
The authors propose a scalable pipeline to teach LLMs how to "read the room" across multiple turns. This involves a multi-LLM collaboration to build a tree-structured preference dataset.
1. Persona Pool Generation
A diverse pool of 3,310 distinct user personas was created. The process started with 20 manually crafted "seed" profiles, which GPT-4o then used to iteratively generate more. To ensure diversity, the researchers used Sentence Transformers to calculate semantic similarity, discarding any new persona that was more than 60% similar to an existing one.
2. Multi-LLM Collaboration for Data
To build a training dataset without expensive human labeling, four LLMs were used in specialized roles:
- Role-playing LLM: Acts as the user based on a specific persona.
- Induction LLM: Analyzes the conversation history to identify which parts of the persona have been "revealed" so far.
- Preferred LLM: Generates a response tailored specifically to those revealed traits.
- Rejected LLM: Generates a generic, non-personalized response.
3. Training and Optimization
The model undergoes a two-stage training process. First, Supervised Fine-Tuning (SFT) is performed on the preferred responses. This is followed by Direct Preference Optimization (DPO) to help the model distinguish between personalized and generic behaviors. Mixing in "agent" data (like CodeActInstruct) was also found to improve the model’s ability to track user preferences over multiple turns.
Evaluation and Results
The authors introduced the ALOE (ALign with custOmized prEferences) benchmark, consisting of 100 carefully curated test cases with unique personas. Key metrics include "Alignment Level" (AL) and "Improvement Rate" (IR) across 10 turns.
- Significant Performance Gains: Models trained with this method showed an average relative improvement of 32.0% in Alignment Level compared to their base versions.
- DPO vs. SFT: Using DPO was significantly more effective than SFT alone. For the OLMo model, incorporating RL/DPO led to a 26.02% performance jump.
- Dynamic Adaptation: Mainstream models like Llama-3 and Mistral have a near-zero Improvement Rate in their base forms—they don't naturally get better at "knowing" the user as the chat progresses. In contrast, the "interact to align" training pushed models like Qwen2-7B to near-perfect alignment scores at later turns.
- Human Validation: The authors compared their GPT-4o "judge" against human annotators and found a Cohen’s Kappa of 0.789, indicating strong agreement.
Conclusion
By treating alignment as a dynamic interaction rather than a static goal, LLMs can provide much more personalized and effective assistance. The "interact to align" framework demonstrates that models can learn to infer and adapt to unspoken user preferences. Future work will explore the boundary where personalization meets safety guardrails and evaluate performance in much longer, "infinite" conversations.
Summary updated