Abstract
The Hearsay system represents a landmark approach to speech recognition by treating "understanding" as a cooperative effort between independent knowledge sources (Acoustic, Syntactic, and Semantic). Using a "hypothesize-and-test" paradigm, the system manages the inherent noise and ambiguity of human speech to achieve close to real-time recognition in task-specific environments like Voice-Chess.
Introduction
In the early 1970s, machine perception of speech was a daunting challenge characterized by high data rates and pervasive errors at every level of analysis—from microphone noise to phonetic ambiguity. Unlike chess or theorem-proving, where humans struggle and "crude" programs are impressive, speech is something humans do effortlessly.
A critical gap existed: traditional "weak" problem-solving methods failed because errors propagate. If a system misidentifies a single phoneme, the entire sentence often collapses. The challenge was to build a system that could use higher-level context (grammar and meaning) to "rescue" lower-level acoustic errors, performing graceful error recovery in a way that mimics human perception.
System Architecture and Methods
The Hearsay system is built on a modular architecture of cooperating independent processes. Its core engine is the hypothesize-and-test paradigm, coordinated by a module aptly named ROVER (Recognition Overlord).
Key components include:
- Knowledge Sources (KS): The system splits expertise into three domains:
- Acoustic-Phonetic: Deals with raw signal parameters (amplitude, zero crossings) and segments them into "pseudo-phonetic" labels.
- Syntactic: Uses "anti-productions" (generated from BNF descriptions) to predict which words are likely to appear before or after a recognized word.
- Semantic: In the Voice-Chess task, this KS knows the board state and only permits hypotheses that represent legal chess moves.
- The Process: One KS (e.g., Syntax) generates an ordered list of hypotheses for an unrecognized portion of the utterance. Other KSs then verify, reject, or re-order these hypotheses based on their own specialized knowledge.
- EAR & SOL: The EAR module handles initial feature extraction, while SOL (System Over Lord) manages communication between the parallel processes.
The Voice-Chess Task
The paper primarily evaluates the system using the Voice-Chess task, where a user speaks a move (e.g., "Bishop moves-to king knight five") and the computer responds. The authors tested the system on seven sets of data featuring five male speakers across four different task domains. The specific example analyzed in depth involves a vocabulary of 31 words and a board state with 46 legal moves.
Key Insights and Results
Modularity and Search Space Reduction
Because each knowledge source is an independent process, a source can be removed or modified without "crippling" the system. This was a precursor to modern microservices and blackboard architectures. Hearsay uses "top-down" semantic knowledge (e.g., "Is this move legal in chess?") to prune the "bottom-up" acoustic search space. Quantitative proof of this cooperation shows that Syntax alone reduces the search space by a factor of 3, but combining it with Semantics yields a factor of 5 reduction.
Performance and Error Recovery
In a sample run of 18 utterances, 15 were recognized correctly, and one was "understood" correctly even if the exact words were slightly off. The system performed recognition in "close to real-time." On a PDP-10 (KA10) processor, the mean computation time per utterance was approximately 10.1 seconds.
The paper demonstrates a case where the system initially favored the incorrect word "king's" over "king" due to acoustic similarities, but eventually recovered the correct path because the subsequent words didn't fit the "king's" hypothesis syntactically.
Limitations and Future Work
The system requires a "neutral phonetic context" table to normalize for different speakers' voices and room acoustics. While the architecture is general, the semantic knowledge is currently hard-coded for specific tasks like chess. Some complex moves required significant computation (up to 48 seconds). The authors view Hearsay as a long-term research tool to gradually incorporate discovered acoustic-phonetic rules that are currently scattered in the literature.
Summary updated