Shallow Semantic Parsing for Spoken Language Understanding

Bonaventura Coppola, Alessandro Moschitti, Giuseppe RiccardiNAACL 2009

TL;DR

This paper presents a novel approach to shallow semantic parsing for spoken language understanding (SLU) by adapting FrameNet-based semantic role labeling (SRL) techniques to conversational speech. They demonstrate that their Support Vector Machine (SVM) and Tree Kernel-based system can be effectively ported to new languages and domains, even with noisy, disfluent spoken data, achieving promising results on both English and Italian datasets.

What Problem Does This Paper Address?

Current Spoken Language Understanding (SLU) systems often rely on simple conceptual annotations and ad-hoc semantic descriptions, limiting their ability to generalize beyond specific applications. While richer semantic models like FrameNet exist, applying them to the complexities of spoken language, characterized by disfluencies and noise, has been challenging. This paper aims to bridge this gap by developing an automatic semantic parser that can leverage FrameNet's rich semantic framework for conversational speech, making SLU systems more robust and adaptable.

Methods

The authors propose a FrameNet-based shallow semantic parsing approach that breaks down the task into four main steps: Target Word Detection, Frame Disambiguation, Boundary Detection (BD), and Role Classification (RC). They employ Support Vector Machines (SVMs) and, crucially, Tree Kernels. Tree Kernels are highlighted as a key innovation, allowing for the modeling of structural features derived from syntactic parse trees. This approach enables greater independence from specific application domains and languages. They compare two feature sets: traditional "standard" SRL features and automatic structural features derived from Tree Kernels. The system is implemented as a combination of specialized SRL labelers, one for each frame.

Datasets

  • English FrameNet 1.3: A large, publicly available dataset of semantically annotated and syntactically parsed sentences. This dataset was used for initial experiments and benchmarking.
  • Italian Spoken Dialog Corpus: A newly introduced benchmark corpus consisting of 50 real human-human dialogs from a help-desk facility. This corpus contains 1,677 annotated frame instances, covering 154 FrameNet frames and 20 new domain-specific frames. It was manually transcribed and includes partially human-validated syntactic analysis.

Key Insights and Ideas

  • Portability of FrameNet-based SRL: The core idea is that FrameNet-based SRL, particularly when leveraging Tree Kernels, can be effectively ported to new languages (Italian) and domains (spoken dialogs) with relatively little effort.
  • Robustness to Disfluencies: The proposed system demonstrates resilience to disfluencies and noise inherent in spoken language, a significant advantage over systems trained solely on clean written text.
  • Effectiveness of Tree Kernels: Tree Kernels prove to be a powerful tool for capturing structural information in syntactic parse trees, leading to improved performance, especially when dealing with new languages and domains where standard features might be less effective.
  • Kernel Combinations are Key: Combining different kernels (e.g., Polynomial Kernel with Tree Kernel) consistently yielded the best performance across experiments.
  • Small Annotations Suffice: A surprisingly small set of FrameNet-like manual annotations can be sufficient for building accurate semantic labelers for target domains in typical dialog systems.

Key Results

  • English FrameNet: The best system achieved an F1 score of 63% for tagging FrameNet data with exact boundaries and role labels. When trained on 90% of the data for BD and RC, the PK+TK kernel achieved an F1 of 0.847 for BD (nodes) and 0.630 for BD+RC (words).
  • Italian Spoken Dialogs: The system achieved an extremely good F1 score on the Italian dialog corpus, outperforming the results on the English FrameNet corpus. The paper highlights that the F1 for the Italian corpus was "extremely good" and comparable to the English results despite the Italian corpus containing disfluencies and being morphologically more complex. The PK+TK combination consistently provided the best results.

Limitations and Future Work

  • Limitations:
    • The current system relies on automatically generated syntactic parses, which can introduce errors.
  • The Italian dialog corpus, while valuable, is relatively small.
  • The system's performance on fully automatic transcription and syntactic parsing needs further investigation.
  • Future Work:
    • Further improving RC models by incorporating FrameNet-specific information like Frame and role inheritance.
  • Developing an effective Frame classifier to automatically select Frame labels.
  • Conducting additional tests with fully automatic transcription and syntactic parsing.

Summary updated

BibTeX
@inproceedings{coppola2009shallow,
  author = {Bonaventura Coppola and Alessandro Moschitti and Giuseppe Riccardi},
  title = {{Shallow Semantic Parsing for Spoken Language Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure