A Multi-Agent Computational Linguistic Approach to Speech Recognition

Michael Walsh, Robert Kelly, Gregory M. P. O'Hare, Julie Carson-Berndsen, Tarek Abu-AmerIJCAI 2003

Abstract

Speech recognition traditionally relies on monolithic algorithms that struggle to bridge the gap between low-level acoustic signals and high-level linguistic constraints. This paper proposes a decentralized alternative by recasting the Time Map model of phonology into a Multi-Agent System (MAS) based on the Belief-Desire-Intention (BDI) architecture. By distributing the computational load across specialized agents—such as Feature Extraction, Segment, and Chart agents—the system achieves robust syllable recognition, intelligent error recovery through constraint relaxation, and a clear separation between linguistic knowledge and processing logic.

Introduction

The primary challenge in speech recognition is the transition from "messy" raw audio signals to structured linguistic rules. Sub-word processing, particularly phonology, is computationally intensive and requires a delicate balance between strict linguistic constraints and the inherent uncertainty of human speech. Current systems often lack the flexibility to handle underspecified or noisy input effectively.

The authors address this by developing an architecture designed to:

  1. Distribute heavy workloads across multiple processors.
  2. Handle "underspecified" input through intelligent extrapolation.
  3. Separate linguistic knowledge from processing logic, making the system easier to adapt to new languages.

Methodology: The Time Map Model and BDI Agents

The proposed system integrates the Time Map model with a BDI multi-agent architecture. The Time Map model utilizes Hidden Markov Models (HMMs) to extract phonological features (e.g., "voiced" or "labial") and employs event logic to map these overlapping features into syllables.

Overview of the Time Map model mapping acoustic signals to phonological and syllable representations.

Unlike simple scripted agents, the BDI agents in this system possess internal mental states: Beliefs (knowledge about the signal), Desires (goals like recognizing a specific phoneme), and Intentions (active plans to achieve those goals). The "team" consists of several specialized agents:

  • Feature Extraction Agents: Parallel workers extracting raw acoustic features.
  • Chart Agent: A linguistic expert using a phonotactic automaton to predict upcoming sounds.
  • Windowing Agent: Manages temporal flow and spawns Segment Agents.
  • Segment Agents: Specialized workers tasked with identifying specific sounds (e.g., [p] or [b]).
The Multi-Agent System architecture showing specialized agents and their internal belief states and communication channels.

Key Insights and Collaborative Reasoning

The multi-agent approach enables several sophisticated behaviors that improve recognition robustness:

  • Collaborative Efficiency: Segment Agents share information to avoid redundancy. For instance, if one agent confirms a "voiced" feature, it broadcasts this belief, allowing other agents needing that feature to skip redundant processing.
  • Constraint Relaxation: If the Chart Agent strongly predicts a sound based on linguistic context, the system can lower the acoustic threshold for recognition, mimicking human-like speech perception.
  • Output Extrapolation: In the presence of noise, the Chart Agent can "fill in the blanks" using its linguistic map to maintain valid syllable structures.
  • Separation of Concerns: Decoupling linguistic rules from search algorithms allows for easier adaptation to different languages by simply swapping the phonotactic automaton.

Results and Implementation

The paper serves as a proof-of-concept for a BDI-based multi-agent system governing a complex linguistic model. While quantitative performance metrics like Word Error Rate (WER) are not the focus, the authors demonstrate that the architecture provides a principled means of distributing computational load. Utilizing the Agent Factory prototyping environment, the system exhibits strong deductive behavior, enabling more sophisticated search space pruning and task allocation than traditional monolithic architectures.

Conclusion and Future Work

While the architecture shows promise for parallel processing and modularity, several areas remain for future exploration. The current study lacks direct quantitative benchmarks against standard systems, and the overhead of agent communication in high-vocabulary, real-time environments needs further investigation. Future work will focus on scaling the system and migrating the modular architecture to other languages and more complex task domains beyond syllable recognition.

Summary updated

BibTeX
@inproceedings{walsh2003multi,
  author = {Michael Walsh and Robert Kelly and Gregory M. P. O'Hare and Julie Carson-Berndsen and Tarek Abu-Amer},
  title = {{A Multi-Agent Computational Linguistic Approach to Speech Recognition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2003},
}
Figure