Data-Driven Execution of Multi-Layered Networks for Automatic Speech Recognition

Renato de Mori, Yoshua Bengio, Régis Cardin

Introduction

In the late 1980s, the field of Automatic Speech Recognition (ASR) faced a significant divide between rule-based expert systems and emerging stochastic models. While human experts could manually define acoustic features like "formant transitions," these rules were difficult to generalize across diverse speakers. Conversely, early neural networks struggled with the high dimensionality of raw audio and limited training data. This 1988 study proposes a hybrid "data-driven" execution strategy, utilizing modular Multi-Layered Networks (MLNs) to automate phonetic feature extraction.

Methodology: Modular Architecture and Property Extractors

The core innovation of this research is the use of specialized Property Extractors (PEs) that feed into MLNs. Unlike standard fixed-window Fourier transforms, these PEs employ variable resolution to capture distinct acoustic events. For instance, high temporal resolution is used for sharp transients like plosive "pops," while high frequency resolution tracks the steady-state resonant frequencies (formants) of vowels.

Architecture of the Multi-Layered Network (MLN) integrated with a variable-resolution Property Extractor.

The system operates through Situation-Based Triggering. Instead of a single monolithic network, the architecture executes specific MLNs only when certain acoustic "situations"—such as energy dips or peaks—are detected. This modularity allows the system to focus its computational resources on relevant segments of the speech signal.

Phonetic Feature Coding

Rather than mapping audio directly to characters, the MLNs are trained to identify universal phonetic features (e.g., voiced/unvoiced, labial, fricative). These features serve as an intermediate representation, which the authors argue improves generalization. The networks were trained using the back-propagation algorithm to minimize the error between predicted features and ground-truth labels.

Experimental Setup

The system was evaluated on the "E-set", a notoriously difficult benchmark consisting of the letters {b, c, d, e, g, k, p, t, v} and the digit {3}. These sounds are challenging because they all rhyme and are distinguished only by subtle, brief consonant sounds at the onset.

The training set included 140 utterances from 70 speakers (35 male, 35 female). Testing was conducted on a separate group of 10 speakers to ensure speaker independence. The computational workload was managed on a VAX 8650 and a SUN 4/280, with recognition times averaging 115 milliseconds per letter.

Key Results and Discussion

The proposed system achieved a 9.5% error rate on the speaker-independent E-set, a state-of-the-art result for the time. Vowel recognition was even more robust, yielding a 3.4% error rate.

Confusion matrix for the speaker-independent E-set recognition task.

As shown in the confusion matrix, the majority of classifications fall on the diagonal, representing correct recognitions. The errors that did occur often involved highly similar pairs, such as /b/ and /v/, which are difficult even for human listeners to distinguish in isolation. The system demonstrated strong performance across both male and female speakers, validating the effectiveness of the Property Extractor approach in normalizing speaker-specific spectral variations.

Conclusion

This work represents a pivotal shift from manual feature engineering toward automated, data-driven learning in ASR. By decomposing the recognition task into modular, situation-triggered networks, the authors demonstrated that neural networks could achieve high accuracy on complex phonetic tasks even with limited data. This modular "mixture of experts" approach prefigured many of the architectural principles found in modern deep learning systems. The authors correctly anticipated that as training data scaled, the performance of such learned feature extractors would continue to improve, a prediction that has been fully realized in the decades since.

Summary updated

Abstract

A set of Multi-Layered Networks (MLN) for Automatic Speech Recognition (ASR) is proposed. Such a set allows the integration of information extracted with variable resolution in the time and frequency domains and to keep the number of links between nodes of the networks small in order to allow significant generalization during learning with a reasonable training set size. Subsets of networks can be executed depending on preconditions based on descriptions of the time evolution of signal energies allowing spectral properties that are significant in different acoustic situations to be learned. Preliminary experiments on speaker-independent recognition of the letters of the E-set are reported. Voices from 70 speakers were used for learning. Voices of 10 new speakers were used for test. An overall error rate of 9.5% was obtained in the test showing that results better than those previously reported can be achieved.
BibTeX
@inproceedings{mori1988data,
  author = {Renato de Mori and Yoshua Bengio and Régis Cardin},
  title = {{Data-Driven Execution of Multi-Layered Networks for Automatic Speech Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1988},
}
Figure