Abstract
We describe a joint model for understanding user actions in natural language utterances. Our multi-layer generative approach uses both labeled and unlabeled utterances to jointly learn aspects regarding utterance's target domain (e.g. movies), intention (e.g., finding a movie) along with other semantic units (e.g., movie name). We inject information extracted from unstructured web search query logs as prior information to enhance the generative process of the natural language utterance understanding model. Using utterances from five domains, our approach shows up to 4.5% improvement on domain and dialog act performance over cascaded approach in which each semantic component is learned sequentially and a supervised joint learning model (which requires fully labeled data).
Introduction
In Spoken Language Understanding (SLU), the standard approach has traditionally been a cascaded pipeline: first identifying the domain (e.g., "Movies"), then the intent or dialog act (e.g., "Find Theater"), and finally extracting specific entities or slots (e.g., "Iron Man 2"). However, this sequential processing suffers from error propagation—a mistake in domain classification inevitably leads to downstream failures. Furthermore, these systems typically demand extensive hand-labeled datasets, which are costly and slow to produce.
This research introduces a joint generative model that simultaneously identifies intent, topic, and entities. By leveraging massive web search query logs as prior knowledge, the model achieves high performance even when labeled training data is limited, effectively reducing the dependency on human labeling.
Methodology: The Multi-Layer Context Model (MCM)
The authors propose the Multi-Layer Context Model (MCM), a hierarchical Bayesian generative model. Unlike pipeline systems, MCM treats domain, dialog act, and slots as interdependent layers within a single hierarchy.
The model follows a top-down generation process where the domain choice influences the dialog act, which subsequently determines the sequence of slots (modeled via a Hidden Markov Model). A key innovation is the use of Informative Priors. By analyzing web search query logs (e.g., Bing) and entity lists (e.g., IMDB), the researchers created non-uniform priors. These provide the model with "hints" about word associations—for instance, the word "room" strongly suggests the "Hotel" domain, while "hotel california" might refer to either a movie or a hotel depending on the context.
Experimental Setup
The model was evaluated using a dataset of 18,084 natural language utterances across five domains: movies, restaurants, hotels, events, and a general "other" category. The data included 42 unique dialog acts and 41 slot tags. External knowledge was incorporated from over 2 million search queries to construct the prior distributions. MCM was benchmarked against traditional cascaded "Sequence-SLU" and "Tri-CRF," a state-of-the-art supervised joint model.
Results and Discussion
The joint modeling approach allows the system to utilize shared context effectively. For example, identifying a movie name entity helps solidify the domain choice, which in turn clarifies the intent.
As shown in the performance metrics, the "WebPrior-MCM" variant significantly outperforms supervised baselines when labeled data is scarce (e.g., at the 10-25% mark). Overall, the MCM approach yielded up to a 4.5% improvement in domain and dialog act accuracy over the cascaded baseline. While discriminative models like CRFs often perform better with abundant data, this generative Bayesian approach proves superior in low-resource scenarios by effectively utilizing unlabeled data.
Conclusion and Future Work
While MCM excels at high-level domain and intent identification, it slightly underperforms supervised models in fine-grained semantic tagging (slot filling). This is attributed to the simpler HMM assumption used for word sequences. Future research aims to integrate CRF-based sequence learners within the Bayesian framework to combine high-level joint reasoning with precise entity extraction.
Summary updated