Abstract: Rashmi Gangadharaiah and Balakrishnan Narayanaswamy (AWS AI) presented a novel approach to Natural Language Understanding (NLU) for task-oriented dialogs at ACL 2020. Their research introduces a recursive, template-based tree decoder designed to handle complex, nested requests that traditional "flat" NLU systems struggle to parse. By representing dialog frames as hierarchical trees rather than linear slot sequences, the model achieves significant performance improvements on complex datasets.
The Limitations of Flat NLU
Most current virtual assistants rely on a flat NLU architecture. In this paradigm, a user's utterance is mapped to a single intent (e.g., book_flight) and a set of independent slots (e.g., from_city: Seattle, to_city: Seoul). While effective for simple queries, this approach fails when faced with recursive or hierarchical language, often referred to as the "Pizza Problem."
Consider the request: "I want two medium veggie pizzas and one small lemonade." A flat system might identify two quantities and two sizes but lacks the structural mechanism to correctly associate each size with its respective item. Developers often attempt to circumvent this by creating complex, concatenated labels like item1_size or item2_quantity. However, this leads to a combinatorial explosion of labels, making models difficult to train and nearly impossible to scale to broader domains.
Proposed Methodology: Recursive Tree Decoding
To address these challenges, the authors propose treating NLU as a hierarchical frame generation task. Their multi-stage neural architecture generates a structured "bracketed" tree representation of the input sentence.
Architecture Overview
- Encoder: The system utilizes BERT to generate contextualized vector representations of the input text.
- Slot Decoder: This component predicts BIO (Begin-Inside-Outside) tags to identify the boundaries of potential slot values within the utterance.
- Recursive Tree Decoder: The core innovation is an LSTM-based decoder that generates a structural template top-down. Instead of predicting a sequence of labels, it produces "non-terminals" (structural nodes like
item) and "terminals" (placeholders for values). - Pointer Network: For every terminal placeholder (denoted by a
$), a pointer network references the original sentence to "copy" the specific word into the appropriate slot. - Global Context Loss: To prevent the decoder from repeating nodes or losing structural coherence, the authors introduced a novel loss function. This uses the BERT
[CLS]token to predict the set of all unique nodes that should appear in the final tree, acting as a global sanity check during training.
Experimental Evaluation
The model was evaluated against two primary benchmarks:
- ATIS (Airline Travel Information System): A standard NLU benchmark. Because ATIS queries are predominantly "flat," it serves as a baseline for standard performance.
- Simulated Ordering Dataset: A new dataset created by the authors to mimic complex retail scenarios. It features nested intents and multiple items with distinct properties, specifically designed to test recursive understanding.
Key Results
The performance gap between traditional flat models and the proposed recursive model becomes evident when complexity increases:
| Dataset | Flat Model (F1) | Recursive Template Model (F1) |
|---|---|---|
| ATIS | 87.51 | 88.01 |
| Simulated Ordering | 32.85 | 97.67 |
On the standard ATIS dataset, the improvement was modest, as there was little hierarchical depth to exploit. However, on the Simulated Ordering dataset, the flat model fundamentally broke down, while the recursive model maintained near-perfect accuracy.
Insights and Future Directions
A key takeaway from this research is the benefit of separating structure from content. By generating a template separately from filling slots via a pointer network, the model learns the underlying logic of a domain independently of specific vocabulary. Furthermore, the hierarchical structure allows the model to share "statistical strength"; for example, different location slots can share a common city_name child node, helping the model learn what constitutes a city across various contexts.
While the results on simulated data are promising, the authors note that real-world human conversation—complete with disfluencies and slang—presents further challenges. Future work will focus on scaling this approach to open-domain scenarios and testing it against more diverse, human-to-human conversation datasets.
Summary updated