A Robust Semantic Frame Parsing Pipeline on a New Complex Twitter Dataset

Yu Wang, Vijay Srinivasan, Hongxia Jin

TL;DR

Researchers have developed a robust semantic frame parsing pipeline designed to handle the "messy" reality of social media text. By combining a novel training data expansion technique with a three-step inference process, their system significantly outperforms state-of-the-art models in identifying intents and "slots" (like locations or dates) within complex, long-form tweets.

What Problem Does This Paper Address?

Most Spoken Language Understanding (SLU) systems are trained on "clean" benchmark datasets like ATIS or SNIPS. These datasets consist of short, predictable queries (e.g., "Play jazz music"). However, when these models face the "wild west" of Twitter, they crumble.

The authors identify two primary culprits for this failure:

  1. Out-of-Distribution (OOD) Patterns: Real-world text often contains irregular structures or "auxiliary" fluff that the model never saw during training.
  2. Out-of-Vocabulary (OOV) Tokens: New words, specific addresses, or unique slang that aren't in the training dictionary.

Current Recurrent Neural Network (RNN) models lack a native mechanism to handle these anomalies, leading to poor performance on long, complex sentences.

Methods

The authors propose a two-pronged strategy to build "robustness" into the parsing pipeline:

1. Expanding the Training Dataset

Instead of just training on raw text, they use delexicalization. This involves replacing specific entities with generic placeholders. For example, "assault on 5/12" becomes <Incident> on <Date>. They generate multiple varieties of each training sentence to cover different OOD and OOV scenarios, teaching the model to recognize the structure of an intent rather than just specific words.

2. A Multistep Inference Algorithm

When the model encounters a new sentence, it doesn't just take a single guess. It follows a three-step process:

  • Step 1: Replace. Unknown words are swapped for a special <unk> token, and known patterns are delexicalized via string matching.
  • Step 2: Expand. This is the clever bit. The system looks at the probability ($p_i$) of neighboring tokens. If a neighbor has a high probability of belonging to a certain slot (above a threshold $T_r = 0.9$), the system "swallows" that token into the delexicalized span.
  • Step 3: Merge. Finally, it merges contiguous <unk> tokens and delexicalized spans if they share the same predicted label, creating a clean, final parse.

Datasets

The study utilizes two main datasets:

  • SNIPS: A standard benchmark for SLU with short, simple queries.
  • New Complex Twitter Dataset: Introduced by the authors, this dataset contains 885 tweets related to public safety incidents (Fire, Crime, Traffic, Natural Disasters).
  • Complexity: It averages 33.2 tokens per tweet, compared to just ~9-11 in standard benchmarks.
  • Volatility: It has a massive OOV rate of 26.5% (vs. 2.7% in SNIPS) and an OOD pattern rate of 23.6% (vs. 5.2% in SNIPS).

Key Insights and Ideas

  • Probability as a Guide: The "aha moment" here is using the model's own uncertainty to define boundaries. By looking at the tag probability of neighboring words, the model can intelligently decide that an unknown word like "Hazardville" is likely part of an <Address> because of the context surrounding it.
  • Structure over Vocabulary: The paper argues that for complex tasks, the semantic frame (the relationship between parts of the sentence) is more important than the specific vocabulary. Delexicalization forces the model to learn these relationships.
  • The "Long-Tail" of Twitter: The authors demonstrate that Twitter isn't just "noisy"—it's structurally different. The high frequency of "auxiliary messages" (contextual fluff) requires a model that can distinguish between the core event and the surrounding noise.

Key Results

The "new pipeline" was tested against three state-of-the-art baselines (Attention BiRNN, Slot-Gated BiRNN, and Bi-model BiLSTM).

  • On the Twitter Dataset: The new pipeline provided a massive boost. For the Bi-model BiLSTM, Intent Accuracy jumped from 73.21% to 81.13%, and the Slot Tagging F1 score rose from 67.12% to 78.16%.
  • On OOD/OOV Specific Data: The gains were even more dramatic. On tweets specifically containing OOD patterns, F1 scores improved by over 30% compared to baseline models.
  • Generalization: Even on the "easy" SNIPS dataset, the pipeline improved performance, showing that robustness doesn't come at the cost of standard accuracy.

Limitations and Future Work

While the paper presents a significant step forward, a few areas remain open:

  • Threshold Sensitivity: The expansion step relies on a predefined threshold ($T_r = 0.9$). The paper doesn't deeply explore how varying this threshold affects different types of text.
  • Lexicon Dependency: The initial "Replace" step still relies on predefined lexicons for string matching. If the initial lexicon is too small, the "Expand" step has less of a foundation to build upon.
  • Computational Overhead: The multistep inference and iterative expansion likely add latency compared to a single-pass RNN, which might be a consideration for real-time applications.

The authors conclude by showcasing a web-based tool that uses this pipeline to parse incident reports, suggesting that this "robustness" is ready for real-world deployment in emergency response and public safety monitoring.

Summary updated

Abstract

Most recent semantic frame parsing systems for spoken language understanding (SLU) are designed based on recurrent neural networks. These systems display decent performance on benchmark SLU datasets such as ATIS or SNIPS, which contain short utterances with relatively simple patterns. However, the current semantic frame parsing models lack a mechanism to handle out-of-distribution (OOD) patterns and out-of-vocabulary (OOV) tokens. In this paper, we introduce a robust semantic frame parsing pipeline that can handle both OOD patterns and OOV tokens in conjunction with a new complex Twitter dataset that contains long tweets with more OOD patterns and OOV tokens. The new pipeline demonstrates much better results in comparison to state-of-the-art baseline SLU models on both the SNIPS dataset and the new Twitter dataset <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Our new Twitter dataset can be downloaded from https://1drv.ms/u/s!AroHb-W6_OAlavK4begsDsMALfE?e=c8f2XX. Finally, we also build an E2E application to demo the feasibility of our algorithm and show why it is useful in real application.
BibTeX
@inproceedings{wang2022robust,
  author = {Yu Wang and Vijay Srinivasan and Hongxia Jin},
  title = {{A Robust Semantic Frame Parsing Pipeline on a New Complex Twitter Dataset}},
  booktitle = {ICKG},
  year = {2022},
  url = {https://openreview.net/forum?id=XlGBvmsYWl},
}

Loading related content...

Figure