Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?

Zhiqiang Hu, Roy Ka-Wei Lee, Nancy F. Chen

TL;DR

Current task-oriented dialogue (TOD) systems struggle to handle impolite user inputs, often failing to understand or respond appropriately. This paper introduces a novel impolite dialogue corpus and demonstrates that existing TOD systems perform significantly worse on impolite conversations. They also propose a data augmentation technique using text style transfer to improve system robustness, showing promising but not complete solutions.

What Problem Does This Paper Address?

Task-oriented dialogue (TOD) systems are designed to help users complete specific tasks, like booking tickets or making reservations. While these systems have become quite sophisticated, they largely operate under the assumption that users will be polite. This is a significant limitation because, in real-world scenarios, users can be impatient, frustrated, or simply impolite, especially when the system makes errors or is slow. The paper highlights that existing TOD systems are not robust to these impolite interactions, leading to poor user experiences. The core problem is the lack of research and datasets addressing how TOD systems handle impolite user utterances, which is crucial for real-world deployment.

Methods

The authors tackle this problem in two main ways:

  1. Corpus Construction: They created a new dataset of impolite dialogues by taking existing polite dialogues from the MultiWOZ 2.2 dataset and having human annotators rewrite user utterances to be impolite. This was done under specific role-playing scenarios (e.g., "imagine you are in a rush and frustrated") to encourage natural impolite language while preserving the original content and ensuring fluency.
  2. Data Augmentation: To improve TOD system performance on impolite dialogues, they proposed a data augmentation strategy. This involves using text style transfer techniques to automatically generate impolite user utterances. They fine-tuned pre-trained language models (T5-large and BART-large) on their impolite corpus to learn the style transfer from polite to impolite, then used these models to generate synthetic impolite data to augment the training set for TOD systems.

Datasets

  • MultiWOZ 2.2: This is the base dataset used for creating the impolite dialogue corpus. The authors note that 99% of its user utterances are classified as polite.
  • Impolite Dialogue Corpus: A newly constructed dataset of 1,573 dialogues and 10,667 impolite user utterances, created by rewriting MultiWOZ 2.2 dialogues. This corpus is made publicly available.

Key Insights and Ideas

  • Politeness Assumption is Unrealistic: The paper strongly demonstrates that the assumption of polite users in TOD systems is a major bottleneck for real-world applicability.
  • Impoliteness Significantly Degrades Performance: Even state-of-the-art TOD systems, when tested on impolite dialogues, show a substantial drop in performance across key metrics like inform, success, and overall combined scores.
  • Data Augmentation Helps, But Isn't a Silver Bullet: While generating impolite data via text style transfer improves system performance on impolite dialogues, it doesn't fully bridge the gap compared to performance on polite dialogues. This suggests that handling impoliteness requires more than just more data.
  • Linguistic Features Matter: The analysis shows that longer impolite utterances and those with lower politeness scores (but not extremely low, which is counter-intuitive) pose greater challenges for TOD systems. This indicates that specific linguistic characteristics of impolite speech are key factors.
  • Abusive Language vs. Subtle Impoliteness: Interestingly, the paper found that systems sometimes performed better on dialogues with overtly abusive language than on those with more subtly impolite or sarcastic utterances.

Key Results

  • Performance Drop: Tested TOD systems showed a 7-15% decrease in their combined score when evaluated on impolite dialogues compared to polite ones.
  • Data Augmentation Improvement: The data augmentation strategy improved the performance of all six tested TOD systems. For instance, PPTOD's combined score increased from 78.4 to 85.2 on impolite dialogues after augmentation.
  • Human Evaluation: Human evaluation confirmed the automatic results, showing a performance gap for impolite dialogues. UBAR and PPTOD, when augmented, showed improvements but still lagged behind their performance on original dialogues.
  • Keyword Analysis: Rewritten dialogues showed a shift from polite terms ("please," "thank you") to direct commands ("find," "get," "give") and expressions of impatience ("hurry," "fast").

Limitations and Future Work

  • Data Augmentation Gap: While data augmentation helps, it doesn't fully resolve the issue. The authors suggest that specialized techniques beyond simple data augmentation might be needed to handle impolite users effectively.
  • Complexity of Impoliteness: The analysis of politeness bins revealed that extremely impolite dialogues (bin 1) were sometimes handled better than moderately impolite ones (bin 2), suggesting a complex relationship that warrants further investigation.
  • Future Work: The authors hope their corpus and findings will encourage more research into designing robust TOD systems capable of handling impolite users. They also suggest exploring ways to distill user intentions from lengthy or complex impolite expressions.

Summary updated

Abstract

Task-oriented dialogue (TOD) systems have assisted users on many tasks, including ticket booking and service inquiries. While existing TOD systems have shown promising performance in serving customer needs, these systems mostly assume that users would interact with the dialogue agent politely. This assumption is unrealistic as impatient or frustrated customers may also interact with TOD systems impolitely. This paper aims to address this research gap by investigating impolite users' effects on TOD systems. Specifically, we constructed an impolite dialogue corpus and conducted extensive experiments to evaluate the state-of-the-art TOD systems on our impolite dialogue corpus. Our experimental results show that existing TOD systems are unable to handle impolite user utterances. We also present a data augmentation method to improve TOD performance in impolite dialogues. Nevertheless, handling impolite dialogues remains a very challenging research task. We hope by releasing the impolite dialogue corpus and establishing the benchmark evaluations, more researchers are encouraged to investigate this new challenging research task.
BibTeX
@article{hu2025are,
  author = {Zhiqiang Hu and Roy Ka-Wei Lee and Nancy F. Chen},
  title = {{Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?}},
  journal = {IEEE Trans. Comput. Soc. Syst.},
  year = {2025},
  doi = {10.1109/tcss.2024.3521020},
}
Figure