Balancing data-driven and rule-based approaches in the context of a Multimodal Conversational System

Srinivas Bangalore, Michael JohnstonNAACL 2004 · OpenReview

Abstract

We address the issue of combining data-driven and grammar-based models for rapid prototyping of a multimodal conversational system. Moderate-sized rule-based spoken language models for recognition and understanding are easy to develop and provide the ability to prototype conversational applications rapidly. However, scalability of such systems is a bottleneck due to the heavy cost of authoring and maintenance of rule sets and inevitable brittleness due to lack of coverage in the rule sets. In contrast, data-driven approaches are robust and the procedure for model building is usually simple. However, the lack of data in an application context limits the ability to build data-driven models, especially in multimodal systems. We also present methods that reuse data from different domains and investigate the limits of such models in the context of an application domain.

Introduction

Building robust multimodal conversational systems often encounters a "cold start" problem. Developers must choose between rule-based systems, which are excellent for rapid prototyping but brittle in real-world use, and data-driven statistical systems, which are robust but require vast amounts of annotated, domain-specific data. This challenge is particularly acute in multimodal interfaces where inputs like speech and pen-based gestures must be processed simultaneously. Collecting and annotating such "speech + ink" data is labor-intensive and expensive.

The MATCH System Architecture

The researchers utilized the MATCH (Multimodal Access To City Help) application as a testbed. MATCH provides a city guide interface where users can interact via speech and gestures (e.g., circling areas on a map).

The MATCH system architecture showing the integration of speech and multimodal components via the MCUBE bus.

The architecture integrates various modules like Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and a Multimodal UI through a central communication bus (MCUBE).

Bridging the Gap: Methodology

The paper explores techniques to build robust models using limited or out-of-domain data by bootstrapping from existing grammars.

1. Bootstrapping Language Models for ASR

To improve ASR without a large corpus, three strategies were tested:

  • Grammar Sampling: Instead of using a rigid hand-crafted grammar, the authors randomly sampled millions of paths to create a synthetic corpus, training a standard N-gram model. This provided greater flexibility than the original rules.
  • Data Migration: A corpus from a software helpdesk domain was "migrated" by replacing domain-specific terms with MATCH-specific classes (e.g., "software" became "cuisine type").
  • Unsupervised Adaptation: Large general corpora (like Switchboard) were adapted using untranscribed audio from the target domain.

2. Robust Multimodal Understanding

Once speech is recognized, the system must extract semantic meaning. The authors compared two robust approaches to map noisy input to valid commands:

  • Pattern Matching (Edit-Distance): Using Finite State Transducers (FSTs) to find the "closest" valid command in the grammar.
  • Classification-based (Adaboost): Treating understanding as a sequence labeling task using IOB (Inside, Outside, Beginning) tagging to identify predicates (intents) and arguments (details).
Finite state representations of speech, gesture, and the resulting integrated meaning.

Key Insights and Results

The study yielded several critical findings regarding the interplay between rules and data:

  • Statistical Rules outperform Rigid Rules: Converting a grammar into a statistical N-gram model via sampling improved word accuracy from 41.6% to 60.6%, demonstrating that while rule structure is valuable, rigidity is a hindrance.
  • Migration is Effective: Migrating linguistic units from a helpdesk domain to a city guide domain provided a significant performance boost over using raw out-of-domain data.
  • Complementary Strengths: The classification approach (Adaboost) excelled at identifying the user's overall intent (84.1% predicate accuracy), while the pattern-matching approach was superior at extracting specific details (arguments), achieving the best Concept String Accuracy (61.3%).

Conclusion

The research demonstrates that it is possible to achieve high performance in multimodal systems by bootstrapping synthetic data from grammars and migrating data from unrelated domains. While the classification-based approach struggled with non-local information, its strength in intent identification suggests a hybrid model—where classification identifies intent and pattern matching fills specific slots—would be optimal for future conversational AI development.

Summary updated

Abstract

We address the issue of combining data-driven and grammar-based models for rapid prototyping of a multimodal conversational system. Moderate-sized rule-based spoken language models for recognition and understanding are easy to develop and provide the ability to prototype conversational applications rapidly. However, scalability of such systems is a bottleneck due to the heavy cost of authoring and maintenance of rule sets and inevitable brittleness due to lack of coverage in the rule sets. In contrast, data-driven approaches are robust and the procedure for model building is usually simple. However, the lack of data in an application context limits the ability to build data-driven models, especially in multimodal systems. We also present methods that reuse data from different domains and investigate the limits of such models in the context of an application domain.
BibTeX
@inproceedings{bangalore2004balancing,
  author = {Srinivas Bangalore and Michael Johnston},
  title = {{Balancing data-driven and rule-based approaches in the context of a Multimodal Conversational System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
  url = {https://openreview.net/forum?id=rybJSQWdbB},
}
Figure