TL;DR
Hearsay II (HSII) is a landmark speech understanding system that introduces a highly modular, parallel architecture based on the "blackboard" model. By treating independent knowledge sources (acoustics, syntax, semantics) as asynchronous processes that collaborate via a global data structure, HSII aims to overcome the error-prone nature of connected speech and scale to complex tasks like news retrieval.
What Problem Does This Paper Address?
Speech understanding is notoriously difficult because the input is "inherently errorful." A single source of information—like just looking at phonemes—isn't enough to decode a sentence reliably. Earlier systems like Hearsay I (HSI) proved that a "hypothesize-and-test" paradigm worked, but they suffered from scalability issues and were difficult to adapt to new domains or parallel hardware.
The authors tackle the challenge of building a system that can handle larger vocabularies (1200+ words), less restricted grammars, and diverse knowledge sources (prosody, user models, etc.) while exploiting the emerging power of multiprocessor architectures.
Methods
The core of Hearsay II is its system organization, which revolves around three main pillars:
- The Blackboard (Global Data Structure): All partial analyses are stored in a generalized 3-dimensional network. The dimensions are:
- Levels of Representation: Ranging from low-level acoustics and phonetics to high-level lexical and syntactic structures.
- Time: The temporal position within the speech utterance.
- Alternatives: Multiple competing hypotheses for the same segment of speech.
- Independent Knowledge Sources (KS): Knowledge is decoupled into modular units. Each KS is a self-activating, asynchronous process. A KS has three roles:
- Triggering: Knowing when it has something useful to contribute based on the current state of the blackboard.
- Hypothesizing: Making an "educated guess" about an aspect of the speech.
- Verifying: Evaluating, reordering, or rejecting hypotheses made by other KSs.
- Parallel Execution: Unlike its predecessor, HSII is designed to run on a 16-processor mini-computer system, allowing different knowledge sources to work on different parts of the problem simultaneously.
Datasets
The paper describes a specific task domain rather than a standardized benchmark dataset (which were rare in 1974):
- Task: Voice-based retrieval of daily wire-service news stories.
- Vocabulary: Approximately 1200 words.
- Input: Connected speech processed via 1/3-octave filter-banks and LPC-derived (Linear Predictive Coding) vocal tract parameters.
Key Insights and Ideas
- The "Blackboard" Metaphor: The most influential idea here is the global workspace. Imagine a group of experts (Knowledge Sources) standing around a blackboard. When one expert sees something they recognize, they write a note. Another expert might see that note and add their own insight. This allows for a "bottom-up" (acoustics to meaning) and "top-down" (semantics to acoustics) flow of information simultaneously.
- Asynchronous Cooperation: By making KSs independent and self-activating, the system doesn't need a rigid, top-down controller. This makes the system robust; if one KS fails or provides a weak hypothesis, others can still provide corrective context.
- Explicit Structural Support: Hypotheses aren't just floating ideas; they are explicitly linked to the evidence that supports them, creating a web of contextual dependencies.
Key Results
As this is a system organization paper, the results are primarily architectural:
- Increased Sophistication: HSII represents a significant jump in generality over HSI, moving from a research prototype to a framework capable of handling complex, real-world tasks.
- Modularity: The authors successfully demonstrated that knowledge (like syntax or phonetics) could be separated from the underlying system machinery, allowing for easy experimentation with different algorithms.
- Hardware Integration: The system was designed to be operational on a custom 16-processor mini-computer at CMU, marking an early foray into parallel AI processing.
Limitations and Future Work
- Development Status: At the time of writing, the system was still "under development" and "expected to be operational." Consequently, this paper lacks final accuracy or latency benchmarks for the full 16-processor implementation.
- Complexity of Coordination: While asynchronous processes offer flexibility, the paper hints at the challenge of "attention focusing"—deciding which KS should run when resources are limited—which remains a complex scheduling problem in blackboard systems.
- Production System Scaling: The semantic model is represented as a production system; how well this scales as the "news of the day" grows in complexity is not fully explored here.
Summary updated