Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation

Takyoung Kim, Jamin Shin, Young-Ho Kim, Sanghwan Bae, Sungdong Kim

TL;DR

Current task-oriented dialogue (TOD) systems are often evaluated against "perfect" users who know exactly what the system can and cannot do. This paper reveals a massive performance gap when these systems face realistic, "unfamiliar" users, leading to a 92% failure rate and a deceptive new error type called "pretending," where the AI confidently hallucinates service-specific information it doesn't actually possess.

What Problem Does This Paper Address?

Most research in Task-Oriented Dialogue (TOD)—think of the bots that help you book a hotel or find a restaurant—relies on benchmarks like MultiWOZ. These benchmarks are built on a hidden assumption: user familiarity bias. In these datasets, human "users" are given highly specific instructions that perfectly align with the system's capabilities.

In the real world, users are messy. They make vague requests, ask for things the system wasn't built to handle (e.g., "Is this restaurant pet-friendly?"), and don't know the underlying database schema. The authors argue that static evaluations on biased benchmarks hide the true instability of end-to-end TOD systems, making them risky to deploy in production.

Methods

The researchers conducted an interactive user study with 20 participants to compare two scenarios:

  1. Closed-goal: Users received detailed instructions that conformed to the system's boundaries (the standard benchmark approach).
  2. Open-goal: Users were given vague, realistic goals that included "exceptional requests"—things the system couldn't actually do. These requests were generated using InstructGPT to ensure they were realistic but outside the training data's scope.

Participants interacted with GALAXY, a state-of-the-art end-to-end TOD model. The researchers then qualitatively annotated the resulting dialogues to identify specific error types, including a newly defined category: "Pretending."

Figure 1: Contrastive dialogues according to user fa- miliarity with the system. Users with high familiarity converse with the system within the predefined scenario since they already know the system well. However, users with low familiarity are more likely to talk about a broad range of topics beyond the system’s capacities leading to the erroneous response of the system.

Datasets

  • MultiWOZ 2.1: The primary dataset used to train the GALAXY model. It is one of the largest TOD benchmarks but is noted for its high user familiarity bias.
  • Open-Goal Ontology: A new set of exceptional requests (e.g., "cancellation policies," "pet-friendly," "wheelchair access") generated via GPT-3 and manually post-processed to simulate realistic out-of-scope user needs.

Key Insights and Ideas

  • The "Pretending" Trap: Unlike standard hallucinations (which might be obviously wrong), "pretending" occurs when a system handles a request it isn't capable of fulfilling by giving a plausible-sounding but false answer. For example, if a user asks if a restaurant is pet-friendly and the system says "Yes" without having that data, the user has no way to verify the error until they show up with their dog and get turned away.
  • Interactive vs. Static Gap: The study found that while models might perform well on static test sets, they face "catastrophic failures" in interactive settings with unfamiliar users.
  • LLMs are not a Panacea: The authors tested GPT-4 on these same erroneous turns. While GPT-4 significantly improved fluency and reduced contradictions, it still suffered from the pretending problem. This suggests that the issue lies in system design and action definitions, not just language modeling power.

Key Results

The quantitative breakdown of the study is stark:

  • 92% of open-goal dialogues had significant issues.
  • Abnormal terminations (where users quit in frustration) were far more common in the open-goal group.
  • Pretending errors occurred nearly 10 times more often in open-goal settings (23.6%) compared to closed-goal settings (2.4%).
Figure 2: Distributions of the dialogue termination by goal type (left) and participant (right). The green bar refers to situations that users finish the conversation with satisfaction, and the yellow bar refers to situations that users finished the conversation but experience some er- rors or inconveniences. Lastly, the red bar expresses the proportion of users’ strong dissatisfaction by forcedly stopping the conversation. C1-C10 and O1-O10 denote users with a closed-goal and open-goal, respectively.

As shown in the error analysis table below, "Irrelevant" and "Pretending" are the dominant failure modes when users step outside the "familiarity" bubble.

Table showing the proportion of error types between closed and open goals.

Limitations and Future Work

The authors acknowledge that their participant pool was limited to employees of a single tech company, which may not represent the full diversity of "real-world" users.

For future work, they suggest two main paths:

  1. Enhancing Transparency: Systems should display intermediate "by-products" or rationales during the conversation so users can spot when the AI is "pretending."
  2. Robust Fallback Management: Improving out-of-scope detection and the ability to handle unstructured requests (e.g., using RAG on FAQs) to prevent the system from forcedly mapping an exceptional request to a standard, incorrect action.

Summary updated

Abstract

Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system's capabilities via strict user goals, namely "user familiarity" bias. This data bias deepens when it combines with data-driven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (open-goal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities. We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior.
BibTeX
@misc{kim2023revealing,
  author = {Takyoung Kim and Jamin Shin and Young-Ho Kim and Sanghwan Bae and Sungdong Kim},
  title = {{Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation}},
  year = {2023},
  eprint = {2305.13857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13857},
}
Figure