Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing

Letian Peng, Jingbo Shang

TL;DR

This paper introduces the Active-Passive-Constraint (APC) score, a fine-grained metric to quantify how faithfully an AI character sticks to its persona. By leveraging this score as a reward signal in Direct Preference Optimization (DPO), the authors demonstrate a significant boost in role-playing quality, ensuring AI agents not only mention relevant facts but also avoid contradicting their background stories.

What Problem Does This Paper Address?

Persona-driven role-playing (PRP) is the art of making LLMs act like specific characters based solely on a set of persona statements. The challenge is faithfulness: AI characters often hallucinate details, ignore their backstory, or contradict themselves.

Until now, evaluating this faithfulness was a "black box" process, usually involving asking a larger LLM (like GPT-4) for a coarse, subjective score. This lacks explainability and doesn't provide a clear signal for optimization. The authors aim to turn this vague concept into a rigorous, mathematically formulated constraint satisfaction problem.

Methods

The core innovation is the APC score. The authors argue that faithfulness depends on two types of constraints:

  1. Active Constraints: When a user query is relevant to a persona statement, the AI must entail that statement in its response.
  2. Passive Constraints: When a query is irrelevant, the AI simply must not contradict that statement.
Figure 1: A presentation of the alignment between APC and human’s view on PRP faithfulness.

Mathematically, the APC score is a sum of Natural Language Inference (NLI) probabilities (entailment, neutral, contradiction) marginalized by query-statement relevance. To make this practical, they used symbolic distillation: they prompted GPT-4 to label thousands of examples and then trained smaller, efficient DeBERTa-V3 models (~300M parameters) to act as relevance and NLI discriminators.

Finally, they used this APC score as a reward function for Direct Preference Optimization (DPO), a technique that fine-tunes models to prefer high-scoring responses over low-scoring ones.

An overview of different PRP methods, including the proposed APC-based DPO.

Datasets

  • Original Characters: To avoid "data contamination" (where the LLM already knows the character from its training data), the authors handcrafted three characters: Alice (introverted guitarist), Bob (rigorous professor), and Eve (secretive spy).
  • Famous Figures: They also tested on six historical/fictional figures (e.g., Spartacus, Cleopatra, Newton) from the existing Character-LLM dataset, with persona statements ranging from 77 to nearly 600.
  • Distillation Data: 8.4K pairs for relevance and 18.9K pairs for NLI, all generated/labeled by GPT-4.

Key Insights and Ideas

  • The "Passive" Penalty: Most role-playing errors aren't just missing facts; they are subtle contradictions of irrelevant facts. The APC score is the first to explicitly penalize these "passive" violations.
  • RAG vs. LCM: The authors found that Retrieval-Augmented Generation (RAG) generally outperforms Long-Context Memory (LCM). As the persona grows, LCM suffers from "lost-in-the-middle" effects, where the model ignores constraints buried in the center of a long prompt.
  • The Danger of Experience Upload (EU): While fine-tuning on generated dialogues (EU) helps, it can actually hurt global faithfulness. By focusing the model on specific statements, it often leads the model to ignore or violate other parts of the persona not present in the training sample.
  • Scaling Laws: There is a "sweet spot" for RAG. Faithfulness peaks when retrieving 5–7 statements; beyond that, the extra context starts to confuse the model.
Scaling rules for retrieved statements and the breakdown of active vs. passive constraint satisfaction.

Key Results

  • Human Alignment: The APC score showed a near-perfect correlation with human evaluation, validating it as a reliable metric.
  • DPO Effectiveness: Applying APC-based DPO improved performance across the board. For the character "Eve," DPO increased the $\Delta APC$ score significantly compared to vanilla RAG or LCM.
  • Qualitative Improvement: In case studies, DPO-tuned models provided much more detailed and character-consistent answers. For example, a Spartacus model without DPO gave vague answers, while the DPO version correctly referenced specific tribal affiliations mentioned in the persona.
Case study showing how DPO refines the model's response to be more specific and faithful.

Limitations and Future Work

  • Efficiency: Calculating the strict APC score requires traversing every single persona statement for every query, which is computationally expensive for characters with thousands of facts.
  • Weighting: Currently, all persona statements are treated as equally important. Future work could introduce "global importance" weights to prioritize core character traits.
  • Model Dependency: The discriminators are distilled from GPT-4, meaning the metric might inherit GPT-4's specific biases or preferences. The authors suggest exploring model-agnostic criteria in the future.

Summary updated

Abstract

Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with all persona statements. Unfortunately, existing faithfulness criteria for PRP are limited to coarse-grained LLM-based scoring without a clear definition or formulation. This paper presents a pioneering exploration to quantify PRP faithfulness as a fine-grained and explainable criterion, which also serves as a reliable reference for optimization. Our criterion first discriminates persona statements into active and passive constraints by identifying the query-statement relevance. Then, we incorporate all constraints following the principle that the AI character's response should be (a) entailed by active (relevant) constraints and (b) not contradicted by passive (irrelevant) constraints. We translate this principle mathematically into a novel Active-Passive-Constraint (APC) score, a constraint-wise sum of natural language inference (NLI) scores weighted by relevance scores. In practice, we build the APC scoring system by symbolically distilling small discriminators from GPT-4 for efficiency. We validate the quality of the APC score against human evaluation based on example personas with tens of statements, and the results show a high correlation. We further leverage it as a reward system in direct preference optimization (DPO) for better AI characters. Our experiments offer a fine-grained and explainable comparison between existing PRP techniques, revealing their advantages and limitations. We further find APC-based DPO to be one of the most competitive techniques for sticking with all constraints and can be well incorporated with other techniques. We then extend the scale of the experiments to real persons with hundreds of statements and reach a consistent conclusion.
BibTeX
@inproceedings{peng2024quantifying,
  author = {Letian Peng and Jingbo Shang},
  title = {{Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=bzPmjmiaz8},
}
Figure