Learning Bayesian Networks for Semantic Frame Composition in a Spoken Dialog System

Marie-Jean Meurs, Fabrice Lefèvre, Renato de MoriNAACL 2009

TL;DR

This paper introduces a novel approach using Dynamic Bayesian Networks (DBNs) to automatically compose semantic frame structures from spoken dialog utterances. This method goes beyond simple concept extraction to capture richer, hierarchical meaning, offering a more robust understanding for dialog systems, especially when dealing with noisy speech input.

What Problem Does This Paper Address?

Spoken Dialog Systems (SDSs) rely on understanding the meaning of user utterances to respond appropriately. Traditional approaches often extract flat sets of concepts or use hand-crafted rules, which can be brittle and struggle with the inherent variability and complexity of human speech. The challenge is to move towards more sophisticated, robust semantic representations that capture the nuanced, hierarchical structure of meaning, enabling dialog managers to make better decisions. This paper specifically addresses the need for a stochastic method to automatically derive these complex, nested semantic structures (frames and frame elements) from speech, going beyond simple keyword spotting or conceptual decoding.

Methods

The core of this work is the application of Dynamic Bayesian Networks (DBNs) for semantic frame composition. The process involves two main decoding steps:

  1. Basic Concept Extraction: First, basic concepts are derived from speech transcriptions.
  2. Semantic Frame Composition: Then, inferences are made on sequential semantic frame structures. This step considers all available information from previous levels (words and concepts) to compose hypothesized "branches" into a single, utterance-span tree.

The authors explore three DBN model variations:

  • Frame/FE Compound Variable: Merging frames and frame elements (FEs) into a single variable.
  • Frame and FE Variables (Joint Decoding): Decoding frames and FEs separately but jointly.
  • Frame and FE Variables (2-Level Decoding): A hierarchical approach where frames are decoded first, and then used as observed variables for FE decoding. This 2-level approach is highlighted as being the most efficient.

To handle ambiguities and hierarchical representations, a "tree-projection algorithm" is used to derive sub-branches associated with concepts. A small set of manually defined rules is also employed to retrieve missing connections lost during the sequential decoding process. The DBN parameters are trained on automatically annotated data derived from a rule-based system.

Datasets

  • MEDIA Corpus: This is the primary dataset used. It's a collection of 1,257 dialogs from a Wizard of Oz system involving negotiation between users and a tourist information server. The corpus is annotated with concepts, specifiers, and importantly, semantic frames and frame elements (FEs). The authors developed a frame ontology for the MEDIA domain, consisting of 21 frames and 86 FEs, based on the FrameNet project's principles.
  • Training Data: Automatically annotated using a rule-based process involving pattern matching and logical rules.
  • Test Set: A subset of 225 speaker turns from the MEDIA corpus, manually annotated for evaluation.

Key Insights and Ideas

  • DBNs for Hierarchical Semantics: The paper demonstrates that DBNs are a powerful tool for learning and composing hierarchical semantic structures (frames and FEs), moving beyond flat concept representations.
  • Stochastic vs. Rule-Based: While rule-based systems can achieve high precision for frame identification, DBNs offer the crucial advantage of providing n-best lists of hypotheses with confidence scores, which is vital for stochastic dialog managers.
  • 2-Level Decoding Efficiency: The 2-level DBN decoding approach offers a good balance between performance and computational efficiency, making it a promising candidate for real-time dialog systems.
  • Bridging ASR and SLU: The proposed method effectively bridges the gap between Automatic Speech Recognition (ASR) output and the semantic understanding required by the dialog manager, even with noisy ASR inputs.

Key Results

The DBN-based systems achieved strong performance on the MEDIA test set. Key results include:

  • High Frame Identification: An average F-measure of 0.95 for frame identification using the rule-based system on the test set, indicating the reliability of the annotation process.
  • Comparable DBN Performance: The three DBN model variations (compound, joint, 2-level) showed comparable performance, with the 2-level system being slightly superior and more efficient.
  • Robustness to Noise: The systems demonstrated good performance even when processing concepts decoded from noisy ASR output (14.8% word error rate, 24.3% concept error rate), achieving F-measures in the range of 0.78-0.84 for frames and FEs.
  • Sub-Frame Link Accuracy: Good results were also obtained for sub-frame links, confirming the ability of DBNs combined with a small rule set to generate consistent hierarchical structures.

Limitations and Future Work

  • Manual Rule Set: While the DBNs are stochastic, a small set of manually defined rules is still used to retrieve lost connections during decoding. Future work could explore fully stochastic methods for this.
  • Dialog Manager Integration: The paper acknowledges that while the DBNs provide rich semantic representations and n-best lists, the impact of these outputs on the dialog manager's decision accuracy needs further assertion.
  • Efficiency of Factored Approach: The paper notes that while factoring frames and FEs increases complexity, it allows for dealing with ambiguities in frame-FE links, suggesting further exploration of factored models.

Summary updated

BibTeX
@inproceedings{meurs2009learning,
  author = {Marie-Jean Meurs and Fabrice Lefèvre and Renato de Mori},
  title = {{Learning Bayesian Networks for Semantic Frame Composition in a Spoken Dialog System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure