TL;DR
This paper introduces novel neural transition systems for understanding complex user requests in virtual agents, moving beyond simple intent/slot extraction to model hierarchical semantic structures. Their approach, particularly the Phrase Structure Shift-Reduce (PS-SR) model, achieves competitive results against state-of-the-art sequence tagging models, demonstrating the effectiveness of structured representations for tasks like product ordering.
What Problem Does This Paper Address?
Current virtual agents often struggle with complex user requests that go beyond single intents and flat lists of entities. Existing Natural Language Understanding (NLU) systems typically extract a "flat" meaning representation, which is insufficient for nuanced interactions like product ordering where multiple intents, entities, and their attributes need to be understood hierarchically. This paper addresses the need for more structured semantic representations that can capture these complex relationships, enabling virtual agents to better understand and fulfill user needs in domains like food ordering.
Methods
The authors propose two novel neural transition-based systems for modeling hierarchical semantic representations:
- Multi-task Shift-Reduce (MT-SR): This model uses multiple independent transition systems, one for each level of the hierarchy, with tree constraints to enforce nesting. It builds representations bottom-up, ensuring higher levels incorporate lower-level predictions.
- Phrase Structure Shift-Reduce (PS-SR): This approach treats the hierarchical annotation as a phrase structure tree and uses a single transition system to predict the entire structure in one pass. It employs a transition system similar to constituency parsers, using a stack to store partial tree structures. The PS-SR model utilizes right-branching, which the authors argue is more suitable for noun-phrase-heavy structures common in product ordering.
Both models leverage deep neural networks, specifically LSTMs, to learn the transition actions based on parser states. For word representations, they combine embeddings from general English text and domain-specific food-related data, using multivariate linear regression to align them. Character-level representations are also incorporated using bidirectional LSTMs.
Datasets
The authors evaluate their models on two custom food services datasets: one for pizza ordering and another for burger ordering. These datasets consist of customer call logs relevant to food ordering, with utterances annotated for intents, entities, attributes, and items. The datasets were split into training, development, and test sets. The paper notes that these datasets were created due to a lack of annotated data for the product ordering domain.
Key Insights and Ideas
- Hierarchical representations are crucial for complex domains: The paper strongly advocates for structured, hierarchical semantic representations over flat ones, especially for tasks like product ordering where relationships between items, attributes, and quantities are vital.
- Transition systems offer a powerful framework: Applying transition-based parsing techniques, traditionally used for syntactic parsing, to semantic role labeling and hierarchical structure extraction proves effective.
- PS-SR's efficiency and effectiveness: The Phrase Structure Shift-Reduce (PS-SR) model, by predicting the entire hierarchical structure in a single pass and leveraging constituency parsing principles, demonstrates superior performance and efficiency compared to the multi-task approach.
- Right-branching for noun phrases: The choice of right-branching in the PS-SR model is highlighted as beneficial for modeling noun phrases, which are common in product ordering and often contain head-final structures in English.
Key Results
The proposed neural transition-based models, particularly the PS-SR model, achieve competitive results against state-of-the-art sequence tagging models like BiLSTM-CRF.
- The PS-SR model (both left and right branching) outperforms the BiLSTM-CRF model on both the Burger and Pizza datasets. For example, on the Burger dataset, PS-SR (RB) achieves an F-score of 83.11%, compared to BiLSTM-CRF's 82.70%.
- The MT-SR model performs slightly lower than BiLSTM-CRF, which the authors attribute to MT-SR's greedy decoding approach versus BiLSTM-CRF's global normalization.
- Right-branching (RB) consistently shows slightly better performance than left-branching (LB) for the PS-SR model across both datasets.
Limitations and Future Work
- Greedy Decoding in MT-SR: The MT-SR model's reliance on greedy decoding limits its performance compared to models with global normalization.
- Annotation Reliability: While the authors report 86% agreement between annotators, this still leaves room for potential noise in the training data.
- Coreference Resolution: The current annotation scheme does not resolve coreference relations, which is identified as a potential area for future work.
- Generalizability: While the annotation scheme aims for abstraction to improve generalizability, further work is needed to test this across a wider range of applications.
- Intra-utterance Coreference: The authors explicitly mention plans to annotate and model intra-utterance coreference relations in future work.
Summary updated