A Comparison of Deep Learning Methods for Language Understanding

Mandy Korpusik, Zoe Liu, James R. GlassInterspeech 2019 · DOI

Introduction

In the field of Spoken Language Understanding (SLU), the primary objective is to parse user utterances into structured semantic representations. This process, known as slot filling or semantic tagging, involves identifying specific entities—such as locations, dates, or cuisines—within a natural language string.

Example of semantic slot tagging for a flight query.

Historically, this task relied heavily on Conditional Random Fields (CRFs) paired with extensive manual feature engineering. This paper benchmarks modern neural architectures—Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Bidirectional Encoder Representations from Transformers (BERT)—to evaluate their effectiveness in eliminating the need for hand-crafted features across diverse domains like air travel and nutrition logging.

Methodology

The study compares four distinct modeling approaches:

  • RNNs: Utilizing bidirectional Gated Recurrent Units (GRUs) to capture sequential dependencies.
  • CNNs: Implementing 1D-CNNs with sliding windows to extract local contextual features.
  • BERT: Leveraging a pre-trained "base" model with a task-specific classification layer. The authors notably kept the BERT parameters "frozen," fine-tuning only the final layer.
  • CRF Baseline: A traditional statistical model using n-grams, POS tags, and lexicons.

These models were tested on the ATIS flight dataset, a restaurant booking corpus, and a challenging meal description dataset containing both written and spoken (ASR-transcribed) entries.

Key Insights: Uncertainty and Robustness

One of the most significant findings involves the relationship between model confidence and prediction accuracy. The researchers observed a strong correlation between the softmax probability scores (confidence) and the resulting F1-scores.

Correlation between model confidence threshold, F-score, and data uncertainty.

As shown in the data, higher confidence thresholds lead to significantly higher F-scores, though at the cost of leaving more data "uncertain." This insight is crucial for production dialogue systems, where low confidence scores can be used to trigger proactive clarification questions rather than providing incorrect information.

Furthermore, the use of "word pieces" in BERT effectively addressed the Out-of-Vocabulary (OOV) problem. By breaking rare terms into smaller chunks, the model can generalize better than traditional word-level models that often fail on unseen vocabulary.

Performance and Efficiency

The results indicate a clear trade-off between accuracy and computational cost:

  • Accuracy: BERT set new state-of-the-art benchmarks on ATIS (98.1 F1) and the Restaurant task (91.6 F1). Neural models also proved more robust than CRFs when dealing with the "noise" of speech recognition errors.
  • Efficiency: Despite its accuracy, BERT is computationally expensive, with 110 million parameters and significant latency. In contrast, the CNN has only 3.1 million parameters and runs efficiently on standard CPUs, making it far more practical for mobile or edge devices.

Conclusion

While BERT provides the highest accuracy through transfer learning, simpler architectures like CNNs remain highly relevant for real-time applications due to their efficiency. Future developments in SLU may focus on hybrid approaches that utilize confidence scores to improve user interaction and explore more efficient versions of large-scale transformer models.

Summary updated

BibTeX
@inproceedings{korpusik2019comparison,
  author = {Mandy Korpusik and Zoe Liu and James R. Glass},
  title = {{A Comparison of Deep Learning Methods for Language Understanding}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1262},
}
Figure