Abstract
Slot-filling and intent detection are the backbone of conversational agents such as voice assistants and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments, we demonstrate that when NATURE operators are applied to the evaluation set of popular benchmarks, the model accuracy can drop by up to 40%.
Introduction
Modern voice assistants (VAs) rely on two core Natural Language Understanding (NLU) tasks: Intent Detection (ID) (e.g., determining if a user wants to play music) and Slot-Filling (SF) (e.g., identifying the specific artist or year). While current deep learning models achieve near-perfect scores on benchmarks like ATIS and SNIPS, these datasets often consist of "too clean," written-style text that fails to account for the messy reality of human speech. The NATURE framework introduces a suite of "natural" perturbations—such as verbal fillers, synonyms, and phonetic slips—to expose a startling "fragility gap" in state-of-the-art models.
Methodology
The NATURE (Natural Alterations of Textual Utterances for Realistic Evaluation) framework applies three categories of transformations to existing evaluation sets to simulate realistic spoken language while preserving semantic meaning:
- Fillers: These are semantically poor but syntactically disruptive markers. They include Beginning-of-Sentence (BOS) and End-of-Sentence (EOS) phrases (e.g., "Actually..." or "...if you please"), as well as Pre- and Post-verb markers (e.g., "like" or "basically").
- Synonymy: Words are replaced with contextually appropriate alternatives. Rather than simple dictionary swaps, a BERT-base model is used to ensure replacements fit the sentence's flow and part-of-speech.
- Speako: This simulates "slips of the tongue" or Speech-to-Text (STT) errors by converting words into the International Phonetic Alphabet (IPA) and swapping them with phonetically similar terms (e.g., changing "watch" to "which").
Experimental Results and Insights
The researchers evaluated NATURE against state-of-the-art architectures, including Stack-Prop+BERT and Bi-RNN, across three pillar benchmarks: ATIS, SNIPS, and NLU-ED. Key findings include:
- Performance Collapse: On a "Hard" evaluation set—where the perturbation causing the most model confusion is selected—End-to-End (E2E) accuracy for the Stack-Prop+BERT model plummeted from 80.6% to 28.9% on average.
- The Position Trap: EOS fillers proved surprisingly disruptive. Adding a simple phrase like "if you please" to the end of a command dropped accuracy by ~40%, suggesting models rely heavily on token position rather than semantic content.
- Adverbs as Shortcuts: Models struggled significantly when adverbs and adjectives were swapped. This indicates that models often use specific, less-diverse word classes as "shortcuts" for intent detection rather than truly understanding the utterance.
- Augmentation Limitations: While training with NATURE-augmented data improved robustness on noisy sets, it decreased performance on original "clean" test sets by approximately 2.3%, highlighting the difficulty of generalizing across different data distributions.
Conclusion
NATURE reveals that current NLU models for voice assistants are highly susceptible to simple, semantically-preserving variations in spoken language. Future work will expand this framework to include multilingual support, specific modeling for speech impediments (e.g., stutters), and multi-turn dialogue scenarios where errors can compound over time.
Summary updated