Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems

Manaal Faruqui, Dilek Hakkani-Tür

Abstract

As more users across the world are interacting with dialog agents in their daily life, there is a need for better speech understanding that calls for renewed attention to the dynamics between research in automatic speech recognition (ASR) and natural language understanding (NLU). We briefly review these research areas and lay out the current relationship between them. In light of the observations we make in this paper, we argue that (1) NLU should be cognizant of the presence of ASR models being used upstream in a dialog system's pipeline, (2) ASR should be able to learn from errors found in NLU, (3) there is a need for end-to-end datasets that provide semantic annotations on spoken input, (4) there should be stronger collaboration between ASR and NLU research communities.

The Traditional Pipeline and Its Failures

For decades, the speech and language research communities have operated in silos. Speech researchers typically focus on perfecting Word Error Rates (WER) at conferences like ICASSP, while NLU researchers focus on semantic parsing and intent classification at venues like ACL. This has led to a rigid "pipeline" approach: an ASR system transcribes audio into text, which is then passed to an NLU system for interpretation.

While this worked for simple, keyword-based search queries, it is increasingly inadequate for modern conversational agents like Alexa, Siri, or Google Assistant. The core issue is that spoken language is fundamentally different from the clean, edited text found on the web. Spoken input is characterized by disfluencies, prosodic cues, and systematic errors introduced by the ASR process itself.

The Complexity of Spoken Language

Spoken language contains nuances that traditional NLU models, trained on typed text, often fail to capture. For instance, a user might say, "book a flight from London to... umm no, from Boston to Denver." A standard NLU model might get confused by the self-correction, whereas a human—or an ASR-aware model—would recognize the disfluency.

Differences between typed and spoken queries.

Furthermore, acoustic features like pauses and pitch can change the entire meaning of a sentence. A pause can distinguish between "No, don't stop" and "No don't... stop." Without access to the underlying audio or ASR confidence scores, the NLU component is essentially working with one hand tied behind its back.

The Divergence Problem

Despite the explosion in smart speaker adoption—with 31% of US broadband households owning one by 2019—the academic gap between speech and language research is actually widening. Analysis of major NLP conferences shows a concerning trend: the percentage of speech-related papers at ACL dropped from 3.3% in 2018 to just 1.8% in 2020.

Declining share of speech-related papers at NLP conferences.

This divergence prevents the development of "ASR-aware" NLU. If NLU models were trained to expect systematic ASR errors (such as confusing "stair light" with "sterilite"), they could use semantic context to recover the user's true intent. Currently, most NLU models treat ASR output as ground truth, leading to brittle systems that fail when the transcription is slightly off.

Proposed Research Paradigms

To bridge this gap, the paper proposes three main research directions:

  1. ASR to NLU Transfer: Instead of passing a single text string, ASR systems should pass "n-best" lists of hypotheses or acoustic features (tone, hesitation) to the NLU. This allows the NLU to consider multiple interpretations of ambiguous audio.
  2. NLU to ASR Transfer: NLU should provide feedback to the ASR. For example, if a user pauses, the Dialog Manager can signal the ASR to keep listening because a specific entity (like a phone number) is still expected. This helps solve the "endpointing" problem where systems cut users off too early.
  3. End-to-End Spoken Language Understanding (SLU): This approach skips the intermediate text transcription entirely, training models to map audio directly to semantic meaning. While promising, these "black-box" models are currently difficult to debug and deploy in production environments compared to modular pipelines.

The Data Bottleneck

A major hurdle to these advancements is the lack of high-quality, natural datasets. Most existing benchmarks are "Frankenstein" datasets created by running Text-to-Speech (TTS) on existing text data. These lack the natural stutters, background noise, and varied accents of real human speech. Furthermore, privacy concerns make it difficult to collect and share the spontaneous, real-world audio needed to train the next generation of robust conversational agents.

Conclusion

The wall between ASR and NLU is a legacy of an era when speech and text were treated as separate problems. To build truly conversational AI, we must move toward joint optimization, where ASR and NLU are aware of each other's uncertainties and work together to decode human intent from the messy reality of spoken language.

Summary updated

Abstract

As more users across the world are interacting with dialog agents in their daily life, there is a need for better speech understanding that calls for renewed attention to the dynamics between research in automatic speech recognition (ASR) and natural language understanding (NLU). We briefly review these research areas and lay out the current relationship between them. In light of the observations we make in this paper, we argue that (1) NLU should be cognizant of the presence of ASR models being used upstream in a dialog system's pipeline, (2) ASR should be able to learn from errors found in NLU, (3) there is a need for end-to-end datasets that provide semantic annotations on spoken input, (4) there should be stronger collaboration between ASR and NLU research communities.
BibTeX
@inproceedings{faruqui2022revisiting,
  author = {Manaal Faruqui and Dilek Hakkani-Tür},
  title = {{Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems}},
  booktitle = {Comput. Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=nH5xoRlb9IE},
}
Figure