Abstract
Xuanli He, Quan Tran, Gholamreza Haffari. Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association. 2019.
This paper introduces a hierarchical neural semantic parser designed for multi-turn dialogues, specifically for an automated email assistant. By combining a Pointer Mechanism to copy user-specific arguments and a Context-Dependent Architecture to track conversation history, the authors achieve a new state-of-the-art accuracy of 73.4% on the Email Assistant dataset.
Introduction
Semantic parsing—the task of converting human language into machine-executable logical forms—is notoriously difficult in dialogue settings. The authors identify two primary hurdles:
- Content vs. Function: User utterances contain both functional words (e.g., "set body to") and content arguments (e.g., "hello world"). Standard sequence-to-sequence (seq2seq) models often struggle to distinguish between the two, frequently "hallucinating" content by trying to generate it from a fixed vocabulary rather than copying it from the input.
- Contextual Dependency: In a conversation, the meaning of a single turn often depends on previous turns. For example, the command "cancel" is ambiguous without knowing the preceding context (e.g., canceling an email draft versus a contact deletion).
Traditional models relied on heavy hand-crafted feature engineering. This paper proposes a purely neural approach to handle both copying and context.
Methodology
The authors build upon a standard attentional seq2seq framework, evaluating both RNN and Transformer variants. Their architecture introduces two key modules:
Pointer Mechanism
Inspired by summarization research, this module uses a "soft switch" ($p_{gen}$). At each decoding step, the model calculates the probability of generating a word from the predefined vocabulary versus copying a word directly from the input utterance based on attention scores. This allows the model to handle specific user strings (like names or subjects) that it may not have seen frequently during training.
Context-Dependent Mechanism
To capture dialogue flow, the authors use a hierarchical architecture. A conversation-level RNN maintains a "global" state ($g_i$) that is updated after every utterance. Furthermore, the encoder processes a concatenation of the current utterance and the previous $k$ utterances, using relative position embeddings to help the model distinguish between the immediate request and older history.
Experimental Results
The study utilizes the Email Assistant dataset, consisting of 4,759 utterances across 113 interactive sessions. The combination of the Transformer backbone, context-awareness, and the pointer mechanism established a new state-of-the-art (SOTA) accuracy of 73.4%, compared to the previous SOTA of 62.3%.
Error Reduction
The pointer mechanism significantly reduced copy-related errors. In the baseline RNN model, incorrect instances dropped from 37 to 18; in the context-aware model, they dropped from 36 to 13. Qualitatively, the context-dependent architecture allowed the model to correctly interpret short, ambiguous commands like "cancel" by linking them to previous actions in the dialogue history.
The "Goldilocks Zone" of History
The authors discovered that incorporating exactly 3 previous utterances yielded the best performance. If the history window is too large (e.g., 5 utterances), the salient information from the current turn gets "contaminated" by irrelevant details from the distant past, leading to a drop in accuracy.
Conclusion
This research demonstrates that a purely neural approach can effectively handle the complexities of dialogue-based semantic parsing. While the pointer mechanism is powerful, it can sometimes be "sticky," struggling to switch back to generation mode. Future work will explore neural denoising mechanisms to filter out noise from longer dialogue histories, potentially allowing for even deeper contextual awareness.
Summary updated