NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue

Iñigo Casanueva, Ivan Vulic, Georgios Spithourakis, Pawel Budzianowski

Abstract: We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intents that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. Finally, we benchmark a series of current state-of-the-art NLU models on NLU++; the results demonstrate the challenging nature of the dataset, especially in low-data regimes, and call for further research on ToD NLU.

Introduction

Most standard Natural Language Understanding (NLU) datasets used in research, such as ATIS or SNIPS, have become "too easy." State-of-the-art models frequently hit 97-98% F1 scores, leaving little room for meaningful evaluation. Furthermore, these datasets often simplify the problem by assuming each user utterance has exactly one intent (e.g., book_flight).

In reality, industrial applications are messy. Users might say, "Hi, can I change my spa reservation for Friday?", which combines a greeting, a request to change something, and a specific service. Traditional datasets would create a single, monolithic label for this specific combination, leading to "label explosion" and making it nearly impossible to reuse data across different domains. NLU++ addresses this lack of lexical diversity and the rigidity of single-label systems by introducing a modular approach.

Methodology: Intent Modules

The core innovation in NLU++ is the concept of intent modules. Instead of monolithic labels, intents are broken down into atomic, reusable components. This modularity allows for combinatorial expressiveness: a model trained on "change" in a banking context can theoretically generalize that concept to a hotel context.

Comparison of traditional monolithic intents versus the modular approach in NLU++.

The ontology distinguishes between domain-specific modules (e.g., overdraft) and generic modules (e.g., affirm, change, why) that appear across different tasks. This design promotes cross-domain reusability, allowing generic modules to achieve high performance even with limited in-domain data.

Dataset Characteristics

The NLU++ dataset consists of 3,080 examples across two domains: BANKING (2,071 examples) and HOTELS (1,009 examples). It features a rich ontology of 62 total intent modules (26 generic) and 17 slots (10 shared). Unlike many crowd-sourced benchmarks, NLU++ was curated and annotated by dialogue NLU experts to ensure high-quality, nuanced labels.

Statistical comparison of NLU++ against standard NLU benchmarks, highlighting its increased complexity and diversity.

As shown in the table above, NLU++ boasts a significantly higher Type-Token Ratio (TTR) and semantic diversity compared to ATIS, SNIPS, or BANKING77. Crucially, it averages over two intents per example, whereas existing datasets are strictly single-intent.

Key Results and Insights

Benchmarking on NLU++ reveals that current models struggle with multi-label, fine-grained tasks. Two main architectures were evaluated:

  1. MLP-Based Baselines: Using fixed sentence encoders (like ConveRT or LaBSE) with a simple Multi-Layer Perceptron classifier.
  2. QA-Based Models: Reformulating NLU as a Question Answering task. For every possible intent, the model is asked: "Is the intent to ask about [INTENT]?"

QA Supremacy: The QA-based model (ROBB-QA) dominated the benchmarks. In the most challenging low-data setup for the Banking domain, it achieved an 80.3% F1, compared to just 58.6% for the best MLP-based encoder (ConveRT). Treating intent detection as a QA problem provides a massive performance boost because the model can leverage the semantic information contained within the intent description rather than treating labels as meaningless integers.

Limitations and Future Work

While QA-based models are accurate, they are computationally expensive, requiring a pass for every possible intent in the ontology. Additionally, NLU++ currently only covers single-turn utterances. Real-world dialogue is multi-turn, and future versions should include system context to help disambiguate user inputs. The authors also suggest moving toward better parsing for canonical values in slot labeling, as current span-based annotations can be brittle.

Summary updated

Abstract

We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intents that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. Finally, we benchmark a series of current state-of-the-art NLU models on NLU++; the results demonstrate the challenging nature of the dataset, especially in low-data regimes, and call for further research on ToD NLU.
BibTeX
@inproceedings{casanueva2022nlu,
  author = {Iñigo Casanueva and Ivan Vulic and Georgios Spithourakis and Pawel Budzianowski},
  title = {{NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=bnkVa8BOO9},
}
Figure