Aligning LLMs with Individual Preferences via Interaction

Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, Heng Ji

Abstract

As large language models (LLMs) demonstrate increasingly advanced capabilities, aligning their behaviors with human values and preferences becomes crucial for their wide adoption. While previous research focuses on general alignment to principles such as helpfulness, harmlessness, and honesty, the need to account for individual and diverse preferences has been largely overlooked, potentially undermining customized human experiences. To address this gap, we train LLMs that can "interact to align", essentially cultivating the meta-skill of LLMs to implicitly infer the unspoken personalized preferences of the current user through multi-turn conversations, and then dynamically align their following behaviors and responses to these inferred preferences. Our approach involves establishing a diverse pool of 3,310 distinct user personas by initially creating seed examples, which are then expanded through iterative self-generation and filtering. Guided by distinct user personas, we leverage multi-LLM collaboration to develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. Finally, we apply supervised fine-tuning and reinforcement learning to enhance LLMs using this dataset. For evaluation, we establish the ALOE (ALign With CustOmized PrEferences) benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations. Experimental results demonstrate the effectiveness of our method in enabling dynamic, personalized alignment via interaction.

Introduction

Current Large Language Model (LLM) alignment typically relies on "one-size-fits-all" principles like helpfulness, harmlessness, and honesty. While these standards make models safer and more reliable, they often result in a "personality-blind" experience. A parent, an artist, and a software engineer might all ask the same question but expect vastly different tones, levels of detail, or types of suggestions. Standard alignment techniques like RLHF often fail to capture these diverse, unspoken preferences because they are not explicitly stated in the prompt.

This work introduces a shift from static alignment toward interactive alignment. We propose that LLMs should treat the initial turns of a conversation as a "discovery phase," cultivating a meta-skill to implicitly infer a user's persona and dynamically tailor subsequent responses. This approach allows the model to align with individual preferences through the natural flow of interaction.

Conceptual overview of interactive alignment where the model implicitly infers user traits (e.g., artist, parent) and increases alignment level over multiple turns.

Methodology: Scalable Persona-Based Alignment

To teach LLMs how to "interact to align," we developed a scalable pipeline that generates high-quality, multi-turn preference data without manual annotation.

1. Persona Pool Construction

We created a diverse pool of 3,310 distinct personas using an iterative self-generation and filtering process. To ensure granular control, we separated "Profiles" (factual data like occupation and location) from "Personalities" (behavioral traits like extroversion or organization). This distinction allows the model to learn both what a user does and how they prefer to communicate.

2. Multi-LLM Collaboration and Data Structure

We utilized four specialized LLM roles to construct the training dataset:

  • Role-playing LLM: Simulates a user based on a specific persona.
  • Induction LLM: Analyzes the conversation history to identify which persona traits have been "revealed" so far.
  • Preferred LLM: Generates a response customized to the inferred persona.
  • Rejected LLM: Generates a generic, safe, but unaligned response.

Unlike standard preference pairs, our data is organized into tree structures. This allows the model to see multiple conversational paths and learn that alignment is a progressive trajectory rather than a single-step task.

Comparison of traditional alignment with the proposed interactive alignment pipeline, featuring multi-LLM collaboration and tree-structured preference data.

Training and Evaluation

We applied a two-stage training recipe: Supervised Fine-Tuning (SFT) on preferred responses to establish the format, followed by Direct Preference Optimization (DPO) to sharpen the model's ability to distinguish between generic and personalized responses.

To measure performance, we established the ALOE Benchmark, a curated set of 100 test cases with unique personas requiring 10-turn interactions. We introduced the Alignment Level (AL) to measure turn-by-turn accuracy and the Improvement Rate (IR) to quantify how much better the model becomes as it gathers more information about the user.

Key Results and Insights

  • Significant Performance Gains: Models trained with the ALOE framework showed an average 32.0% relative improvement in Alignment Level across 10 turns compared to base models.
  • The Necessity of DPO: While SFT helps with conversational flow, DPO was essential for teaching the model to actively prefer nuanced, persona-aligned responses over safe but generic ones.
  • Alignment as a Trajectory: Our results confirm that interaction is a powerful source of information. As the conversation goes deeper, the alignment level of the assistant's response for each round increases, validating the "interact to align" concept.
  • Human-AI Agreement: Automated evaluations using GPT-4o as a judge were validated by human annotators, yielding a high Cohen’s Kappa of 0.789, indicating that our metrics accurately reflect human perceptions of personalization.

Conclusion

By moving beyond static, generalized alignment, we enable LLMs to become more intuitive and personalized partners. The "interact to align" framework demonstrates that models can learn to read between the lines of a conversation, inferring user needs and adapting their behavior dynamically. This meta-skill is a crucial step toward truly personalized AI assistants that respect and reflect the diversity of human preferences.

Summary updated

Abstract

As large language models (LLMs) demonstrate increasingly advanced capabilities, aligning their behaviors with human values and preferences becomes crucial for their wide adoption. While previous research focuses on general alignment to principles such as helpfulness, harmlessness, and honesty, the need to account for individual and diverse preferences has been largely overlooked, potentially undermining customized human experiences. To address this gap, we train LLMs that can ''interact to align'', essentially cultivating the meta-skill of LLMs to implicitly infer the unspoken personalized preferences of the current user through multi-turn conversations, and then dynamically align their following behaviors and responses to these inferred preferences. Our approach involves establishing a diverse pool of 3,310 distinct user personas by initially creating seed examples, which are then expanded through iterative self-generation and filtering. Guided by distinct user personas, we leverage multi-LLM collaboration to develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. Finally, we apply supervised fine-tuning and reinforcement learning to enhance LLMs using this dataset. For evaluation, we establish the ALOE (ALign With CustOmized PrEferences) benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations. Experimental results demonstrate the effectiveness of our method in enabling dynamic, personalized alignment via interaction.
BibTeX
@inproceedings{wu2025aligning,
  author = {Shujin Wu and May Fung and Cheng Qian and Jeonghwan Kim and Dilek Hakkani-Tur and Heng Ji},
  title = {{Aligning LLMs with Individual Preferences via Interaction}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=KbefiqpW7r},
}
Figure