Abstract
Traditional personalized dialogue systems primarily rely on Next Token Prediction (NTP), which often fails to maintain global persona consistency. This paper introduces the Persona-Aware Alignment Framework (PAL), a novel approach that treats persona consistency as an alignment objective. By integrating persona-aware multi-task learning with Direct Preference Optimization (DPO), PAL enables models to distinguish between generic and personalized responses. Experimental results demonstrate that PAL significantly enhances persona consistency, allowing smaller open-source models like Llama 3.1 8B to outperform larger closed-source models such as GPT-4o mini in character-driven dialogue tasks.
1. Introduction
Generating dialogue that remains consistent with a predefined persona (e.g., "I work as an attorney," "I love dogs") is a significant challenge for Large Language Models (LLMs). While standard training objectives like Next Token Prediction ensure grammatical fluency, they often neglect the "global" semantic goal of staying in character.
Two primary hurdles hinder effective personalization:
- Quantification: The difficulty of mathematically scoring how well a response aligns with a specific persona trait.
- Noise: Persona profiles often contain multiple traits, most of which are irrelevant to any single turn in a conversation. These irrelevant traits act as noise, distracting the model from the pertinent context.
2. Methodology
The PAL framework moves beyond simple language modeling by explicitly optimizing for persona alignment through a three-component architecture: Persona-aware Learning, Persona Alignment, and a specialized inference strategy.
2.1 Stage 1: Persona-aware Learning
In the first stage, the model undergoes multi-task training to increase its sensitivity to persona traits. This involves two simultaneous sub-tasks:
- Dialogue-Informed Persona Selection: The model learns to identify the most relevant trait from a persona profile given the current conversation history.
- Persona-Enhanced Dialogue Generation: The model generates a response conditioned specifically on the selected relevant trait.
2.2 Stage 2: Persona Alignment via DPO
To teach the model the semantic difference between a generic response and a personalized one, the authors employ Direct Preference Optimization (DPO). Since standard datasets lack "rejected" examples, the authors construct them synthetically: the human ground-truth response serves as the "preferred" example, while a response generated by the model without access to the persona profile serves as the "rejected" example. This contrastive learning forces the model to prioritize persona-specific information.
2.3 Inference: Select then Generate
At runtime, PAL employs a "Select then Generate" strategy. Rather than processing the entire persona profile at once—which can lead to hallucinations or trait confusion—the model first explicitly selects the most relevant trait and then uses that anchor to generate the final response.
3. Experimental Evaluation
The framework was evaluated on two major benchmarks:
- PERSONA-CHAT (English): Consisting of over 65,000 training samples, including "Revised" versions to test the model's ability to handle rephrased traits rather than just memorizing keywords.
- Baidu-Persona-Chat (Chinese): A large-scale dataset with over 350,000 samples used to validate the framework's cross-lingual robustness.
4. Results and Discussion
PAL demonstrated substantial improvements across both automated and human metrics. On the PERSONA-CHAT dataset, the C.score (a metric for persona consistency) for Llama 3.1 8B jumped from 0.611 to 0.909.
In human evaluations, the PAL-enhanced Llama 3.1 8B achieved a persona consistency score of 2.82/3.0, significantly higher than GPT-4o mini (2.09/3.0) and Gemini Flash (2.13/3.0). Qualitative analysis confirmed that PAL-trained models were far less likely to "hallucinate" traits (e.g., claiming a different profession) compared to base LLMs.
5. Conclusion
The Persona-Aware Alignment Framework shifts the focus of personalized dialogue from simple prediction to explicit semantic alignment. By filtering persona noise and using contrastive optimization, PAL allows even relatively small models to maintain high levels of character consistency. Future work will explore the framework's performance in long-term dialogues where personas may evolve or require deeper contextual memory.
Summary updated