Semantic role labeling with discriminative feature selection for spoken language understanding

Chao-Hong Liu, Chung-Hsien WuInterspeech 2009

TL;DR

This paper tackles the challenge of understanding spoken language, particularly when Automatic Speech Recognition (ASR) errors and ungrammatical speech make traditional methods struggle. They introduce a discriminative feature selection approach for Semantic Role Labeling (SRL) that significantly improves intent classification accuracy, even with noisy ASR output.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) aims to decipher the meaning behind spoken words. A key component of SLU is Intent Classification, which identifies the user's goal. While Semantic Role Labeling (SRL) – identifying who did what to whom – has shown promise in boosting intent classification, applying it to spoken language is fraught with difficulties. State-of-the-art ASR systems often have recognition rates below 80%, and spoken language itself is frequently ungrammatical and contains fragments. These issues make it hard for SRL systems, which rely on structured text, to perform accurately, thus hindering overall SLU.

Methods

The authors propose a novel approach to Semantic Role Labeling (SRL) that combines discriminative feature selection with new "Bernoulli event features."

  • Discriminative Feature Selection: This method aims to select a subset of features that not only represent the original data well but also minimize misclassification errors. It iteratively adds features that maximize a "discriminative likelihood" function, which balances the likelihood of the model with its loss. This is crucial because traditional machine learning methods can degrade in performance when faced with the millions of features often generated in SRL.
  • Bernoulli Event Features: To better handle ASR errors and ungrammaticality, they introduce Bernoulli event features. These features represent sequences of words and their corresponding Part-of-Speech (POS) tags, up to length three. This approach is inspired by N-gram metrics used in machine translation and aims to capture more robust linguistic patterns even in noisy input.

The overall system architecture for intent classification is iterative, incorporating SRL results and using "mapping rules" derived from expert knowledge to group predicate/argument pairs into semantic intents.

Datasets

  • CoNLL-2005 SRL corpus: Used for training and evaluating the Semantic Role Labeling performance.
  • ATIS-3 (Air Travel Information System) corpus: Used for evaluating the intent classification performance, particularly in a semi-supervised setting. This corpus is derived from the Penn Treebank-3.

Key Insights and Ideas

  • Feature Selection is Crucial for SRL: The paper strongly emphasizes that simply increasing the number of features doesn't guarantee better performance in SRL. Instead, selecting a discriminative subset of features is key to achieving optimal results, as shown in their analysis of feature count effects.
  • Bernoulli Event Features Mitigate ASR Noise: The introduction of Bernoulli event features, which capture word and POS sequences, proves effective in improving intent classification even when dealing with ASR errors and ungrammatical speech. This is a significant step towards more robust SLU.
  • SRL Enhances Intent Classification: The study reinforces the idea that incorporating Semantic Role information can significantly improve the accuracy of Intent Classification, even when the SRL is performed on imperfect ASR output.

Key Results

  • Intent Classification Improvement: The proposed approach, using discriminative feature selection and Bernoulli event features, improved intent classification by 3.4%.
  • SRL Performance: The method also showed improvements in Semantic Role Labeling performance.
  • Feature Contribution: Analysis of structural features revealed that while some features had a negative impact on specific roles (likely due to insufficient data), the four structural features (Path, ApexPath, Position, and Bernoulli) generally improved the performance of most other semantic roles.

Limitations and Future Work

  • Data Sparsity for Certain Roles: The paper acknowledges that some semantic roles (e.g., A3, AM-DIR, AM-LOC, AM-NEG, R-A2, R-AM-LOC) showed poor performance, likely due to insufficient training and test samples for these specific roles.
  • Computational Complexity: While feature selection addresses the issue of a huge number of features, the process itself can still be computationally intensive. The paper mentions that search strategies are often used to manage this complexity.
  • Future Work: The authors suggest that further research could explore more sophisticated feature selection methods and potentially investigate how to better handle the data sparsity issues for less frequent semantic roles. The effectiveness of Bernoulli event features for intent classification in a semi-supervised approach on the ATIS corpus also points towards future directions in semi-supervised SLU.

Summary updated

BibTeX
@inproceedings{liu2009semantic,
  author = {Chao-Hong Liu and Chung-Hsien Wu},
  title = {{Semantic role labeling with discriminative feature selection for spoken language understanding}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure