Understanding How People Rate Their Conversations

Alexandros Papangelis, Nicole Chartier, Pankaj Rajan, Julia Hirschberg, Dilek Hakkani-Tur

Abstract

User ratings play a significant role in spoken dialogue systems. Typically, such ratings tend to be averaged across all users and then utilized as feedback to improve the system or personalize its behavior. While this method can be useful to understand broad, general issues with the system and its behavior, it does not take into account differences between users that affect their ratings. In this work, we conduct a study to better understand how people rate their interactions with conversational agents. We specifically focus on agreeableness and extraversion as variables that may explain variation in ratings and therefore provide a more meaningful signal for training or personalization. To elicit these traits, we designed and validated a fictional story grounded in psychology. Our results suggest that extraversion may play a role in user ratings, but more data is needed for significance. Agreeableness, on the other hand, plays a statistically significant role: users who are more agreeable are more likely to provide a higher rating for their interaction. In addition, we found that users who opted to hear the story were, in general, more likely to rate their conversational experience higher than those who did not.

Introduction: The Subjectivity of User Ratings

In the development of Spoken Dialogue Systems (SDS), user ratings—often on a 1-to-5 star scale—serve as the primary metric for evaluating performance. Developers typically average these ratings across all users to determine the success of model updates. However, this approach assumes that all users utilize the rating scale identically.

In reality, a "3-star" rating from a critical, introverted user may reflect a different level of satisfaction than the same rating from a highly agreeable user. Failing to account for these individual differences risks optimizing AI for an "average user" who does not exist, potentially embedding personality-based biases into the system. The challenge lies in measuring user personality without disrupting the conversational flow with lengthy psychological surveys.

Methodology: The Story-Based Survey

To address this, researchers developed a "Story Method" to elicit personality traits naturally during interaction. Instead of a traditional questionnaire, the agent narrates a story about attending a party. At specific points in the narrative, the agent asks questions designed to probe specific traits. For example, asking about the enjoyment of parties targets Extraversion, while asking if the user would check on a sad individual targets Agreeableness.

The conversational agent's story-based utterances designed to elicit personality traits.

This method was validated using Amazon Mechanical Turk, where participants took both the story-based survey and a standard Big Five personality test. The story scores proved to be statistically significant predictors of traditional survey scores ($p < 0.01$). For deployment, the story was implemented as a Finite State Machine (FSM) within an experimental conversational agent to ensure consistency across user interactions.

Results: How Personality Biases Feedback

Analysis of 760 real-world interactions revealed several key insights into how personality influences user feedback:

  • Agreeable Users are "Easy Graders": Agreeableness was a strong predictor of higher ratings ($\beta = 0.316, p = 0.02$). This suggests that perceived improvements in AI performance might sometimes be artifacts of the personality distribution of the user base.
  • The Opt-in Effect: Users who chose to hear the story rated the agent significantly higher (3.63 vs. 3.0) than those who declined. This indicates a selection bias: users open to engagement and storytelling are predisposed to view the agent more favorably.
  • Extraversion's Nuanced Role: While extraversion correlates with human-to-human communication perception, its effect on AI ratings only neared significance ($p = 0.098$). This may be because conversational agents typically lead the dialogue, reducing the need for the AI to satisfy an extrovert's preference for social dominance.
Distribution of Agreeableness and Extraversion scores among the study participants.

Overall, Agreeableness and Extraversion together explained a small but significant portion of the variance in conversational ratings ($R^2 = 0.045, p = 0.012$).

Discussion and Limitations

While personality is a clear factor in user ratings, it only accounts for approximately 4.5% of the variance. The remaining 95% is likely driven by the agent's actual performance, latency, or the user's immediate mood. The study also observed a "ceiling effect" for Agreeableness, as many users who opted into the story already scored high on this trait, which may have limited the observed range of behaviors.

Future research could explore more dynamic, personalized storytelling that adapts to user responses while maintaining the consistency required for scientific measurement. Ultimately, these findings serve as a reminder for researchers: understanding who is providing the ratings is as crucial as the ratings themselves.

Summary updated

Abstract

User ratings play a significant role in spoken dialogue systems. Typically, such ratings tend to be averaged across all users and then utilized as feedback to improve the system or personalize its behavior. While this method can be useful to understand broad, general issues with the system and its behavior, it does not take into account differences between users that affect their ratings. In this work, we conduct a study to better understand how people rate their interactions with conversational agents. One macro-level characteristic that has been shown to correlate with how people perceive their inter-personal communication is personality. We specifically focus on agreeableness and extraversion as variables that may explain variation in ratings and therefore provide a more meaningful signal for training or personalization. In order to elicit those personality traits during an interaction with a conversational agent, we designed and validated a fictional story, grounded in prior work in psychology. We then implemented the story into an experimental conversational agent that allowed users to opt-in to hearing the story. Our results suggest that for human-conversational agent interactions, extraversion may play a role in user ratings, but more data is needed to determine if the relationship is significant. Agreeableness, on the other hand, plays a statistically significant role in conversation ratings: users who are more agreeable are more likely to provide a higher rating for their interaction. In addition, we found that users who opted to hear the story were, in general, more likely to rate their conversational experience higher than those who did not.
BibTeX
@inproceedings{papangelis2021understanding,
  author = {Alexandros Papangelis and Nicole Chartier and Pankaj Rajan and Julia Hirschberg and Dilek Hakkani-Tur},
  title = {{Understanding How People Rate Their Conversations}},
  booktitle = {IWSDS},
  year = {2021},
  url = {https://openreview.net/forum?id=lVwgvAPI25},
}
Figure