Automatic Evaluation of Neural Personality-based Chatbots

Yujie Xing, Raquel Fernández

Introduction

As conversational AI evolves, the focus has shifted from mere functional accuracy to stylistic consistency. "Personality-based" models aim to provide chatbots with a distinct "voice"—whether that of a specific character or a set of psychological traits. However, evaluating these models remains a significant challenge. Traditional metrics like BLEU focus on word overlap rather than stylistic nuance, and human evaluation is prohibitively expensive and slow.

Researchers from the University of Amsterdam propose a novel solution: an automatic evaluation framework that utilizes a statistical "personality recognizer" to assess the linguistic markers of personality in AI-generated text.

Problem Statement

Standard neural chatbots often suffer from a "vanilla" response problem, producing safe but generic output that lacks character. While conditioning models on speaker identities or personality scores (such as the Big Five) can mitigate this, measuring the success of these interventions is difficult. This paper addresses the need for a scalable, objective metric to determine if a chatbot is truly expressing its assigned personality traits.

Methodology

The study evaluates two primary neural architectures based on the Sequence-to-Sequence (SEQ2SEQ) with Attention framework:

  1. The Speaker Model: This model learns unique embeddings for specific individuals, such as characters from popular sitcoms.
  2. The Personality Model: Instead of individual IDs, this model is conditioned on numerical scores for the "Big Five" (OCEAN) traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

The OCEAN-based Evaluation Framework

To evaluate these models, the researchers employed an external tool, the Mairesse personality recognizer, to extract OCEAN scores from the generated responses. They then trained a Support Vector Machine (SVM) classifier to identify characters based solely on these extracted scores. If the classifier can accurately distinguish between characters using only personality markers, it serves as evidence that the chatbot has successfully internalized and expressed those traits.

Datasets

The models were trained using two primary sources:

  • OpenSubtitles: A large-scale dataset of over 50 million movie dialogue lines used for general language pre-training.
  • TV Series Transcripts: Dialogue from Friends and The Big Bang Theory provided a "gold standard" for 13 distinct characters, each with at least 2,000 dialogue turns. This dataset allowed for the mapping of specific characters to their respective personality profiles.

Results and Discussion

The evaluation framework successfully validated that personality traits are a viable signal for character identification. On the original "gold" transcripts, the method achieved F1 scores between 61% and 69%.

Performance comparison of Speaker and Personality models against Gold transcripts and Baseline.

Model Comparison

The "Speaker Model" generally outperformed the "Personality Model" in capturing character nuances. For The Big Bang Theory, the Speaker Model achieved an F1 score of 0.47, compared to 0.30 for the Personality Model. This suggests that individual identity encompasses more complexity than can be captured by five scalar personality values alone.

Synthetic Personalities and Generalization

A key finding was the Personality Model's ability to generalize. When tasked with generating "extreme" synthetic personalities (e.g., maximum Extraversion) that were not present in the training data, the model successfully extrapolated appropriate linguistic styles.

Evaluation of synthetic personality types using the proposed OCEAN-based recognizer.

As shown in the synthetic evaluation results, the model achieved an average F1 score of 0.53 across different traits, significantly outperforming the baseline. This demonstrates that the model learned a robust relationship between OCEAN scores and stylistic markers, even in cases of data extrapolation.

Conclusion and Future Work

The proposed method offers a promising path toward automated, style-aware evaluation of chatbots. However, several challenges remain. Neural models still tend to produce generic responses (e.g., "I don't know"), requiring researchers to filter frequent outputs to isolate personality signals. Furthermore, the method's accuracy is tied to the underlying personality recognizer. Future research should explore more diverse datasets and refined recognizers to enhance the robustness of this evaluation framework across different genres and languages.

Summary updated

BibTeX
@inproceedings{xing2018automatic,
  author = {Yujie Xing and Raquel Fernández},
  title = {{Automatic Evaluation of Neural Personality-based Chatbots}},
  booktitle = {INLG},
  year = {2018},
  doi = {10.18653/v1/w18-6524},
}
Figure