Abstract
We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences, introducing and validating the idea of intent modules that can be combined into complex intents that convey complex user goals, combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intent modules that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. Finally, we benchmark a series of current state-of-the-art NLU models on NLU++; the results demonstrate the challenging nature of the dataset, especially in low-data regimes, the validity of 'intent modularisation', and call for further research on ToD NLU.
Introduction: The Gap in Modern NLU
Most standard Natural Language Understanding (NLU) datasets used in research, such as ATIS or SNIPS, are increasingly considered "solved." State-of-the-art models frequently achieve F1 scores in the 97-98% range. However, these metrics often fail to reflect the messy reality of industrial task-oriented dialogue (ToD) systems.
In production environments, models face several hurdles that academic benchmarks overlook:
- Single-Intent Assumption: Traditional datasets assume one intent per utterance, yet real users often pack multiple requests into a single sentence (e.g., "I want to change my booking and also ask about the spa").
- Low Lexical Diversity: Crowdsourced datasets often suffer from repetitive, simple phrasing that lacks the linguistic variety of real human interaction.
- Rigid Ontologies: Flat, domain-specific intent labels make it difficult to reuse data or logic across different applications.
NLU++ was designed specifically to bridge this gap, providing a high-quality, expert-annotated dataset that mirrors the combinatorial complexity of real-world dialogue.
The NLU++ Dataset and Intent Modularization
NLU++ consists of 3,080 examples spanning two primary domains: BANKING and HOTELS. The dataset includes 62 intent modules (26 generic, 36 domain-specific) and 17 slots with both span-based and canonical value annotations. Unlike many research sets, NLU++ was annotated and filtered by dialogue NLU experts to ensure natural complexity.
The core innovation of NLU++ is Intent Modularization. Rather than using traditional, monolithic labels like change_restaurant_booking, the dataset breaks intents down into atomic modules: change, restaurant, and booking. This approach offers three distinct advantages:
- Combinatorial Expressiveness: A small set of modules can represent a vast array of complex user goals.
- Generalization: Models can learn to recognize "cancel" and "account" separately, allowing them to interpret "cancel account" even if that specific combination was absent from the training set.
- Cross-Domain Reusability: The ontology is split into generic modules (e.g.,
greet,affirm,date) that apply universally, and domain-specific ones (e.g.,overdraft,room_amenities).
Benchmarking Methodology
To evaluate the difficulty of NLU++, the authors benchmarked two distinct architectural paradigms:
- MLP-Based Models: Standard classifiers built on top of frozen, high-performance sentence encoders such as ConveRT or LaBSE.
- QA-Based Models: A newer paradigm that reformulates intent detection as a Question Answering task. For every possible intent, the model is prompted: "Is the intent to ask about [INTENT]?"
Key Insights and Results
Dataset Complexity
NLU++ is objectively more challenging than its predecessors. It features a significantly higher Type-Token Ratio (TTR) and greater semantic diversity. For instance, its TTR is 0.268, compared to just 0.043 for ATIS, indicating a much richer vocabulary and less repetition.
The Superiority of QA-NLU
Treating NLU as a QA problem proved remarkably effective. By leveraging the internal knowledge of large language models (like RoBERTa) through natural language prompts, these models "understand" the semantic meaning of the labels themselves rather than treating them as abstract IDs. This led to significant performance gains:
- Intent Detection (ID): In a challenging 20-fold cross-validation setup, the QA-based model (ROBB-QA) achieved an 80.3 F1 on the Banking domain, whereas the best MLP-based model (LM12-1B) reached only 59.1 F1.
- Slot Labeling (SL): The QA approach also outperformed specialized models like
ConvEx, showing gains of over 20 F1 points in low-data Banking scenarios.
Transferability and Resilience
The split between generic and domain-specific modules was validated through cross-domain testing. Models trained on generic intents in one domain successfully identified them in another with high performance (up to 79.3 F1). Furthermore, QA-based models demonstrated impressive resilience in "low-data" regimes (50-100 examples), a common scenario when bootstrapping new industrial bots.
Limitations and Future Directions
While NLU++ advances the state of the art, several challenges remain:
- Inference Latency: The QA-based approach is computationally expensive. Because the model must evaluate a prompt for every possible intent/slot in the ontology, latency increases linearly with the size of the ontology.
- Single-Turn Focus: The current dataset focuses on single-turn understanding. Future iterations should incorporate multi-turn context to handle nuances like anaphora (e.g., "the savings one").
- Value Extraction: The authors suggest that moving beyond span-based annotation toward more robust canonical value extraction methods will be necessary for complex entities like dates and times.
Conclusion
NLU++ provides a rigorous new benchmark for the NLU community, moving away from "solved" datasets toward the complexity of real-world task-oriented dialogue. By validating the intent modularization approach and demonstrating the power of QA-based NLU, this work sets a new standard for building and evaluating generalizable, industrial-strength dialogue systems.
Summary updated