Benchmarking Natural Language Understanding Services for building Conversational Agents

Xingkun Liu, Arash Eshghi, Pawel Swietojanski, Verena Rieser

Introduction

The proliferation of conversational agents, from smart home assistants to customer service bots, has made Natural Language Understanding (NLU) a cornerstone of modern AI development. NLU systems are tasked with two primary functions: Intent Classification—determining the user's goal (e.g., "set an alarm")—and Named Entity Recognition (NER)—extracting specific parameters (e.g., "7:00 AM"). Despite the availability of numerous commercial and open-source NLU toolkits, developers often lack independent, empirical data to guide their choice of platform. This study addresses this gap by providing a comprehensive, third-party benchmark of four leading NLU services: Google Dialogflow, Microsoft LUIS, IBM Watson Assistant, and the open-source Rasa.

Methodology

The evaluation compared the performance of these services using a rigorous 10-fold cross-validation approach. The services tested include:

  • Commercial Platforms: Google Dialogflow, Microsoft LUIS, and IBM Watson Assistant.
  • Open-Source Framework: Rasa (version 0.10.5, utilizing the spacy_sklearn pipeline).

Due to a training set limitation of 10,000 instances in Microsoft LUIS at the time of testing, the researchers utilized a representative sub-corpus of approximately 11,000 utterances for the primary evaluation. Performance was quantified using standard information retrieval metrics: Precision, Recall, and the F1-score.

The NLU-Evaluation-Data Dataset

A significant contribution of this research is the introduction of the NLU-Evaluation-Data dataset, a large-scale benchmark designed to reflect real-world linguistic diversity. The dataset comprises 25,716 user utterances across 21 distinct domains, including IoT, music, transport, and takeaway orders. It features 64 unique intents and 54 entity types.

Data was collected via Amazon Mechanical Turk to ensure the inclusion of "noisy," informal, and ambiguous language typical of human-robot interaction. For instance, an utterance like "play space invaders please" requires the system to distinguish whether "space invaders" refers to a song, a game, or a movie based on context.

Table 1: Examples of annotated utterances from the NLU-Evaluation-Data dataset across various scenarios and intents.

Comparative Performance Analysis

The benchmarking results revealed distinct strengths and weaknesses across the four platforms. While all services performed competitively in intent classification, significant disparities emerged in entity recognition.

Table 2: Performance metrics (Precision, Recall, and F1-score) for Intent Classification and Entity Recognition across the four NLU services.

Intent Classification

IBM Watson Assistant emerged as the "Intent King," achieving the highest F1-score of 0.882. Statistical analysis confirmed that Watson significantly outperformed its competitors in this category ($p < 0.05$). Rasa, Dialogflow, and LUIS followed closely, with F1-scores ranging from 0.855 to 0.864.

Entity Recognition and the "Watson Paradox"

The results for entity recognition told a different story. While Microsoft LUIS achieved the highest Entity F1-score (0.777), IBM Watson suffered a significant performance drop. Despite its prowess in intent detection, Watson’s entity recognition precision was remarkably low at 0.354. This "Watson Paradox" suggests that the service was overly aggressive in identifying entity candidates, leading to a high volume of false positives. In contrast, Rasa maintained a high precision of 0.859, making it one of the most reliable choices for entity extraction.

Discussion and Limitations

The study highlighted several critical challenges currently facing NLU services:

  1. Contextual Awareness: A major finding was the lack of effective dialogue context utilization. Most services treat utterances as isolated events, struggling with elliptical sentences or fragments (e.g., responding "Yes" to a prompt).
  2. Multi-Intent Utterances: None of the evaluated services supported the detection of multiple intents within a single sentence (e.g., "Turn off the lights and play music").
  3. The Impact of ASR: As this study relied on text-based input, it does not account for the errors introduced by Automatic Speech Recognition (ASR) in voice-first interfaces, which can further degrade NLU accuracy.

Conclusion

For developers seeking a balanced NLU solution, Rasa and Microsoft LUIS offer the most consistent performance across both intent and entity tasks. While IBM Watson is superior for pure intent classification, its current entity recognition logic may require additional filtering or custom logic to mitigate its high false-positive rate. As these cloud services continue to evolve, ongoing independent benchmarking remains essential for the conversational AI community.

Summary updated

Abstract

We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.
BibTeX
@inproceedings{liu2019benchmarking,
  author = {Xingkun Liu and Arash Eshghi and Pawel Swietojanski and Verena Rieser},
  title = {{Benchmarking Natural Language Understanding Services for building Conversational Agents}},
  booktitle = {IWSDS},
  year = {2019},
  doi = {10.1007/978-981-15-9323-9\_15},
}
Figure