The Hearsay Speech Understanding System: An Example of the Recognition Process

Dabbala Rajagopal Reddy, Lee D. Erman, R. D. Fenneli, Richard B. NeelyIJCAI 1973

Abstract

The Hearsay system represents a landmark approach to speech recognition by treating "understanding" as a cooperative effort between independent knowledge sources (Acoustic, Syntactic, and Semantic). Using a "hypothesize-and-test" paradigm, the system manages the inherent noise and ambiguity of human speech to achieve close to real-time recognition in task-specific environments like Voice-Chess.

Introduction

In the early 1970s, machine perception of speech was a daunting challenge characterized by high data rates and pervasive errors at every level of analysis—from microphone noise to phonetic ambiguity. Unlike chess or theorem-proving, where humans struggle and "crude" programs are impressive, speech is something humans do effortlessly.

A critical gap existed: traditional "weak" problem-solving methods failed because errors propagate. If a system misidentifies a single phoneme, the entire sentence often collapses. The challenge was to build a system that could use higher-level context (grammar and meaning) to "rescue" lower-level acoustic errors, performing graceful error recovery in a way that mimics human perception.

System Architecture and Methods

The Hearsay system is built on a modular architecture of cooperating independent processes. Its core engine is the hypothesize-and-test paradigm, coordinated by a module aptly named ROVER (Recognition Overlord).

The Hearsay system architecture, featuring the System Overlord and Recognition Overlord coordinating independent knowledge sources.

Key components include:

  • Knowledge Sources (KS): The system splits expertise into three domains:
  • Acoustic-Phonetic: Deals with raw signal parameters (amplitude, zero crossings) and segments them into "pseudo-phonetic" labels.
  • Syntactic: Uses "anti-productions" (generated from BNF descriptions) to predict which words are likely to appear before or after a recognized word.
  • Semantic: In the Voice-Chess task, this KS knows the board state and only permits hypotheses that represent legal chess moves.
  • The Process: One KS (e.g., Syntax) generates an ordered list of hypotheses for an unrecognized portion of the utterance. Other KSs then verify, reject, or re-order these hypotheses based on their own specialized knowledge.
  • EAR & SOL: The EAR module handles initial feature extraction, while SOL (System Over Lord) manages communication between the parallel processes.

The Voice-Chess Task

The paper primarily evaluates the system using the Voice-Chess task, where a user speaks a move (e.g., "Bishop moves-to king knight five") and the computer responds. The authors tested the system on seven sets of data featuring five male speakers across four different task domains. The specific example analyzed in depth involves a vocabulary of 31 words and a board state with 46 legal moves.

Analysis of the utterance 'Bishop moves-to king knight five,' showing the mapping from acoustic signal to phonetic labels and words.

Key Insights and Results

Modularity and Search Space Reduction

Because each knowledge source is an independent process, a source can be removed or modified without "crippling" the system. This was a precursor to modern microservices and blackboard architectures. Hearsay uses "top-down" semantic knowledge (e.g., "Is this move legal in chess?") to prune the "bottom-up" acoustic search space. Quantitative proof of this cooperation shows that Syntax alone reduces the search space by a factor of 3, but combining it with Semantics yields a factor of 5 reduction.

Performance and Error Recovery

In a sample run of 18 utterances, 15 were recognized correctly, and one was "understood" correctly even if the exact words were slightly off. The system performed recognition in "close to real-time." On a PDP-10 (KA10) processor, the mean computation time per utterance was approximately 10.1 seconds.

The paper demonstrates a case where the system initially favored the incorrect word "king's" over "king" due to acoustic similarities, but eventually recovered the correct path because the subsequent words didn't fit the "king's" hypothesis syntactically.

Limitations and Future Work

The system requires a "neutral phonetic context" table to normalize for different speakers' voices and room acoustics. While the architecture is general, the semantic knowledge is currently hard-coded for specific tasks like chess. Some complex moves required significant computation (up to 48 seconds). The authors view Hearsay as a long-term research tool to gradually incorporate discovered acoustic-phonetic rules that are currently scattered in the literature.

Summary updated

BibTeX
@inproceedings{reddy1973hearsay,
  author = {Dabbala Rajagopal Reddy and Lee D. Erman and R. D. Fenneli and Richard B. Neely},
  title = {{The Hearsay Speech Understanding System: An Example of the Recognition Process}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1973},
}
Figure