Persona-Aware Alignment Framework for Personalized Dialogue Generation

Guanrong Li, Xinyu Liu, Zhen Wu, Xinyu Dai

Abstract

Traditional personalized dialogue systems primarily rely on Next Token Prediction (NTP), which often fails to maintain global persona consistency. This paper introduces the Persona-Aware Alignment Framework (PAL), a novel approach that treats persona consistency as an alignment objective. By integrating persona-aware multi-task learning with Direct Preference Optimization (DPO), PAL enables models to distinguish between generic and personalized responses. Experimental results demonstrate that PAL significantly enhances persona consistency, allowing smaller open-source models like Llama 3.1 8B to outperform larger closed-source models such as GPT-4o mini in character-driven dialogue tasks.

1. Introduction

Generating dialogue that remains consistent with a predefined persona (e.g., "I work as an attorney," "I love dogs") is a significant challenge for Large Language Models (LLMs). While standard training objectives like Next Token Prediction ensure grammatical fluency, they often neglect the "global" semantic goal of staying in character.

Two primary hurdles hinder effective personalization:

  1. Quantification: The difficulty of mathematically scoring how well a response aligns with a specific persona trait.
  2. Noise: Persona profiles often contain multiple traits, most of which are irrelevant to any single turn in a conversation. These irrelevant traits act as noise, distracting the model from the pertinent context.
Example of persona-consistent (check) vs. inconsistent (cross) dialogue responses based on a provided persona profile.

2. Methodology

The PAL framework moves beyond simple language modeling by explicitly optimizing for persona alignment through a three-component architecture: Persona-aware Learning, Persona Alignment, and a specialized inference strategy.

The complete Persona-Aware Alignment (PAL) framework, illustrating the two-stage training process and the 'Select Then Generate' inference strategy.

2.1 Stage 1: Persona-aware Learning

In the first stage, the model undergoes multi-task training to increase its sensitivity to persona traits. This involves two simultaneous sub-tasks:

  • Dialogue-Informed Persona Selection: The model learns to identify the most relevant trait from a persona profile given the current conversation history.
  • Persona-Enhanced Dialogue Generation: The model generates a response conditioned specifically on the selected relevant trait.

2.2 Stage 2: Persona Alignment via DPO

To teach the model the semantic difference between a generic response and a personalized one, the authors employ Direct Preference Optimization (DPO). Since standard datasets lack "rejected" examples, the authors construct them synthetically: the human ground-truth response serves as the "preferred" example, while a response generated by the model without access to the persona profile serves as the "rejected" example. This contrastive learning forces the model to prioritize persona-specific information.

2.3 Inference: Select then Generate

At runtime, PAL employs a "Select then Generate" strategy. Rather than processing the entire persona profile at once—which can lead to hallucinations or trait confusion—the model first explicitly selects the most relevant trait and then uses that anchor to generate the final response.

3. Experimental Evaluation

The framework was evaluated on two major benchmarks:

  • PERSONA-CHAT (English): Consisting of over 65,000 training samples, including "Revised" versions to test the model's ability to handle rephrased traits rather than just memorizing keywords.
  • Baidu-Persona-Chat (Chinese): A large-scale dataset with over 350,000 samples used to validate the framework's cross-lingual robustness.

4. Results and Discussion

PAL demonstrated substantial improvements across both automated and human metrics. On the PERSONA-CHAT dataset, the C.score (a metric for persona consistency) for Llama 3.1 8B jumped from 0.611 to 0.909.

In human evaluations, the PAL-enhanced Llama 3.1 8B achieved a persona consistency score of 2.82/3.0, significantly higher than GPT-4o mini (2.09/3.0) and Gemini Flash (2.13/3.0). Qualitative analysis confirmed that PAL-trained models were far less likely to "hallucinate" traits (e.g., claiming a different profession) compared to base LLMs.

5. Conclusion

The Persona-Aware Alignment Framework shifts the focus of personalized dialogue from simple prediction to explicit semantic alignment. By filtering persona noise and using contrastive optimization, PAL allows even relatively small models to maintain high levels of character consistency. Future work will explore the framework's performance in long-term dialogues where personas may evolve or require deeper contextual memory.

Summary updated

Abstract

Personalized dialogue generation aims to leverage persona profiles and dialogue history to generate persona-relevant and consistent responses. Mainstream models typically rely on token-level language model training with persona dialogue data, such as Next Token Prediction, to implicitly achieve personalization, meaning that these methods tend to neglect the given personas and generate generic responses. To address this issue, we propose a novel Persona-Aware Alignment Framework (PAL), which directly treats persona alignment as the training objective of dialogue generation. Specifically, PAL employs a two-stage training method including Persona-Aware Learning and Persona Alignment, equipped with an easy-to-use inference strategy Select then Generate, to improve persona sensitivity and generate more persona-relevant responses at the semantics level. Through extensive experiments, we demonstrate that our framework outperforms many state-of-the-art personalized dialogue methods and large language models.
BibTeX
@article{li2025persona,
  author = {Guanrong Li and Xinyu Liu and Zhen Wu and Xinyu Dai},
  title = {{Persona-Aware Alignment Framework for Personalized Dialogue Generation}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=C2E69HAwEg},
}
Figure