Abstract
Most recent semantic frame parsing systems for spoken language understanding (SLU) are designed based on recurrent neural networks. These systems display decent performance on benchmark SLU datasets such as ATIS or SNIPS, which contain short utterances with relatively simple patterns. However, the current semantic frame parsing models lack a mechanism to handle out-of-distribution (OOD) patterns and out-of-vocabulary (OOV) tokens. In this paper, we introduce a robust semantic frame parsing pipeline that can handle both OOD patterns and OOV tokens in conjunction with a new complex Twitter dataset that contains long tweets with more OOD patterns and OOV tokens. The new pipeline demonstrates much better results in comparison to state-of-the-art baseline SLU models on both the SNIPS dataset and the new Twitter dataset. Finally, we also build an E2E application to demo the feasibility of our algorithm and show why it is useful in real application.
Introduction
Spoken Language Understanding (SLU) systems are the backbone of modern conversational AI, yet they are often brittle when removed from the controlled environments of benchmark datasets. While models perform well on "clean" datasets like ATIS or SNIPS—which consist of short, predictable queries—they frequently fail when applied to the "wild west" of social media. Social media text, particularly on platforms like Twitter, is characterized by long-form sentences, irregular structures, and a high frequency of novel terms.
This research addresses the failure of current Recurrent Neural Network (RNN) models to handle two primary challenges: Out-of-Distribution (OOD) patterns (irregular structures or "auxiliary fluff") and Out-of-Vocabulary (OOV) tokens (slang, specific addresses, or new entities). We propose a robust parsing pipeline that utilizes a novel training expansion technique and a multi-step inference algorithm to bridge this gap.
The Challenge of Social Media Text
Standard SLU benchmarks are often too simplistic to prepare models for real-world deployment. To demonstrate this, we introduced a New Complex Twitter Dataset consisting of 885 tweets related to public safety incidents (Fire, Crime, Traffic, and Natural Disasters).
Comparing this dataset to the standard SNIPS benchmark reveals a stark contrast in complexity:
- Length: Tweets average 33.2 tokens, compared to approximately 9–11 tokens in standard benchmarks.
- Volatility: The Twitter dataset features an OOV rate of 26.5% (vs. 2.7% in SNIPS) and an OOD pattern rate of 23.6% (vs. 5.2% in SNIPS).
Traditional models crumble under this complexity because they lack a native mechanism to generalize beyond their training dictionary and specific sentence structures.
Proposed Methodology
Our approach builds robustness into the parsing pipeline through a two-pronged strategy involving data delexicalization and a specialized inference process.
1. Training Data Expansion
Instead of training solely on raw text, we employ delexicalization. This process replaces specific entities with generic placeholders (e.g., "Assault on 5/12" becomes <Incident> on <Date>). By generating multiple variations of each training sentence, we force the model to prioritize the underlying semantic frame—the relationship between parts of the sentence—over specific vocabulary.
2. A Three-Step Inference Algorithm
When the model encounters a new utterance, it processes it through a robust three-step pipeline:
- Replace: Unknown words are converted to a special
<unk>token, and known patterns are delexicalized via string matching against existing lexicons. - Expand: The system analyzes the probability ($p_i$) of neighboring tokens. If a neighbor has a high probability (above a threshold $T_r = 0.9$) of belonging to a specific slot, the system "swallows" that token into the delexicalized span. This allows the model to use context to identify OOV tokens (e.g., recognizing an unknown street name because it follows "located at").
- Merge: Contiguous
<unk>tokens and delexicalized spans are merged if they share the same predicted label, resulting in a clean, final semantic parse.
Experimental Results
We evaluated our pipeline against three state-of-the-art baselines: Attention BiRNN, Slot-Gated BiRNN, and Bi-model BiLSTM.
The results on the Twitter dataset were significant. For the Bi-model BiLSTM, Intent Accuracy improved from 73.21% to 81.13%, and the Slot Tagging F1 score rose from 67.12% to 78.16%. The gains were most dramatic on data specifically containing OOD patterns, where F1 scores improved by over 30% compared to baseline models. Notably, the pipeline also improved performance on the "easier" SNIPS dataset, suggesting that robustness does not require a trade-off with standard accuracy.
Conclusion and Future Work
This research demonstrates that by focusing on structural patterns and utilizing model uncertainty (probability thresholds), we can create SLU systems capable of navigating complex, real-world text. While the current pipeline relies on a predefined threshold ($T_r$) and initial lexicons, it provides a significant leap forward for applications in public safety and emergency response monitoring. Future work will focus on reducing the computational overhead of the multi-step inference and exploring dynamic thresholding to further improve generalization across diverse text types.
Summary updated