A Bottom-Up Stepwise Knowledge-Integration Approach to Large Vocabulary Continuous Speech Recognition Using Weighted Finite State Machines

Sabato Marco Siniscalchi, Torbjørn Svendsen, Chin-Hui Lee

Abstract

Traditional Large Vocabulary Continuous Speech Recognition (LVCSR) systems have long relied on monolithic Hidden Markov Models (HMMs), which often act as "black boxes" and require extensive task-specific data. This paper proposes a modular, bottom-up framework that decouples acoustic attribute detection from linguistic decoding. By combining a bank of phonetic attribute detectors with Weighted Finite State Machines (WFSMs), the authors demonstrate a system that is both flexible and high-performing. On the Wall Street Journal benchmark, the proposed approach achieves a Word Error Rate (WER) of 6.0%, significantly outperforming standard hybrid HMM/ANN systems and proving the viability of knowledge-integrated, stepwise decoding for large-scale tasks.

Introduction

For decades, Hidden Markov Models (HMMs) have been the cornerstone of speech recognition. However, these systems are increasingly seen as reaching a performance ceiling due to their integrated nature, which makes it difficult to incorporate explicit phonetic knowledge or optimize components independently. Traditional HMM-based systems typically require massive amounts of transcribed data to train tightly coupled acoustic and language models.

The challenge lies in creating a more modular architecture that can leverage "acoustic-phonetic" knowledge—such as identifying whether a sound is a nasal or a fricative—and scale to large vocabularies. Previous attempts at such modularity often faced computational bottlenecks, particularly memory exhaustion when representing entire utterances as complex state machines. This work addresses these issues through a stepwise knowledge-integration framework using WFSMs.

Methodology

The proposed system, referred to as DET-SYS, moves away from the integrated HMM approach toward a three-stage pipeline that integrates knowledge incrementally.

High-level architecture of the detection-based LVCSR system, showing the flow from input speech to the final 1-best word sequence.

1. Detection-based Frontend

The process begins with a bank of 21 Multi-Layer Perceptrons (MLPs), each acting as an independent "attribute detector." These detectors are trained to identify specific phonetic features, such as vowel, stop, dental, or voiced. The outputs of these detectors are concatenated and processed by a "Merger" MLP, which generates posterior probabilities for 40 distinct phoneme classes.

Detailed architecture of the Detection-based Frontend, illustrating how input MFCCs are transformed into phone posteriors through attribute detectors and an evidence merger.

2. WFSM Composition and the "Sausage" Representation

The system represents the speech signal as a "feature acceptor" ($F$), a WFSM that takes the form of a linear sequence of phoneme posteriors, often called a "sausage" due to its structure. This acceptor is then mathematically composed with other WFSMs representing different layers of knowledge:

  • D (Duration): Enforces minimum durations for phonemes.
  • L (Lexicon): Maps sequences of phonemes to words.
  • G (Grammar): Incorporates language model constraints.

3. Stepwise Pruning

To manage the potential state-space explosion during composition ($F \circ D \circ L \circ G$), the system employs pruning at every stage. This "early pruning" is effective because the initial phoneme estimates from the detection-based frontend are highly accurate, allowing the search space to be narrowed early without losing the optimal path.

Experimental Evaluation

The system was evaluated on the Wall Street Journal (WSJ0) corpus, a standard benchmark for speaker-independent LVCSR.

  • Training Data: The SI-84 set, comprising approximately 15.3 hours of speech from 84 speakers.
  • Evaluation Data: The Nov92 evaluation set, consisting of 330 utterances from 8 speakers.
  • Task: A 5,000-word vocabulary recognition task.

Key Results

The DET-SYS framework demonstrated strong performance compared to established baselines:

System Word Error Rate (WER)
GMM-SYS (Standard HMM) 5.5%
ANN-SYS (Hybrid HMM/ANN) 8.3%
DET-SYS (Proposed) 6.0%

The proposed system achieved a 27.7% relative improvement over the standard hybrid HMM/ANN approach. While it slightly trailed the highly optimized GMM-HMM system, the results validate the modular, detection-based approach as a competitive alternative to traditional monolithic modeling.

Discussion and Future Work

The success of this bottom-up approach highlights several key insights:

  • Decoupling: Separating attribute detection from decoding allows for independent improvement of the frontend without retraining the entire system.
  • Interpretability: Using 21 distinct phonetic attributes provides a richer and more interpretable intermediate representation than raw audio or abstract HMM states.
  • Feasibility: Stepwise integration proves that modular LVCSR is computationally feasible if pruning is applied intelligently throughout the composition process.

Future research will focus on incorporating context-dependency (e.g., triphone modeling) at the attribute level and developing more sophisticated, automated pruning strategies. Additionally, testing the framework on larger vocabularies and more spontaneous speech will be crucial for assessing its robustness in real-world scenarios.

Summary updated

Abstract

A bottom-up, stepwise, knowledge integration framework is proposed to realize detection-based, large vocabulary continuous speech recognition (LVCSR) with a weighted finite state machine (WFSM). The WFSM framework offers a flexible architecture for different types of knowledge network compositions, each of them can be built and optimized independently. Speech attribute detectors are used as an intermediate block to obtain phoneme posterior probabilities over which a phoneme recognition network is designed. Lexical access and syntax knowledge integration over this phoneme network are then performed to deliver the decoded sentences. Experimental evidence illustrates that the proposed system outperforms several hybrid HMM/ANN systems with different configurations on the Wall Street Journal task while it is competitive with conventional LVCSR technology.
BibTeX
@inproceedings{siniscalchi2011bottom,
  author = {Sabato Marco Siniscalchi and Torbjørn Svendsen and Chin-Hui Lee},
  title = {{A Bottom-Up Stepwise Knowledge-Integration Approach to Large Vocabulary Continuous Speech Recognition Using Weighted Finite State Machines}},
  booktitle = {Interspeech},
  year = {2011},
  doi = {10.21437/interspeech.2011-351},
}
Figure