TL;DR
Hearsay-II introduced the blackboard architecture, a landmark framework for coordinating diverse, independent "knowledge sources" to solve complex problems characterized by high uncertainty. By treating speech understanding as an opportunistic search across multiple levels of abstraction—from acoustic segments to semantic intent—it pioneered a modular approach to AI that remains influential in system design today.
What Problem Does This Paper Address?
Speech understanding is notoriously difficult because uncertainty exists at every stage of the process. As shown in Figure 1, a speaker's intention is transformed through layers of semantics, syntax, and phonetics into an acoustic wave, picking up noise and variability along the way.
Traditional linear processing (e.g., first find all phonemes, then all words) fails because an error at a lower level propagates upward, often leading to a combinatorial explosion of possibilities. The authors sought a system that could integrate multiple types of knowledge—acoustic, lexical, syntactic, and semantic—to resolve these ambiguities cooperatively and efficiently.
Methods
The core innovation of Hearsay-II is the Blackboard Architecture. The system consists of three main components:
- The Blackboard: A global, shared database partitioned into levels of abstraction (Parameter, Segment, Syllable, Word, Phrase, etc.). It acts as a "common room" where hypotheses are posted.
- Knowledge Sources (KSs): Independent, modular programs that specialize in specific tasks (e.g., a "Word Spotter" or a "Syntactic Parser"). KSs do not call each other; instead, they are "condition-action" pairs. They monitor the blackboard for specific patterns (the condition) and, when triggered, modify or create new hypotheses (the action).
- The Heuristic Scheduler: Because many KSs might be triggered at once, a scheduler prioritizes actions based on the "credibility" of the hypotheses involved and the potential value of the proposed action.
This setup enables opportunistic problem solving. The system can work "bottom-up" (building words from sounds) and "top-down" (predicting words based on grammatical expectations) simultaneously. It can also perform "island driving," starting from a highly confident word in the middle of a sentence and expanding outward.
Datasets
The system was evaluated on a 1011-word vocabulary task focused on retrieving computer science abstracts.
- Grammar: A context-free semantic grammar (X05) with a "static branching factor" of 10 (meaning, on average, 10 words could follow any given word).
- Test Data: 23 "blind" utterances (sentences the system had never "heard" during development) spoken by a single male speaker in a noisy environment (>65 dB).
Key Insights and Ideas
- Decoupling Knowledge from Control: By forcing KSs to communicate only through the blackboard, the authors created a highly modular system. You could add or remove a grammar module without rewriting the acoustic module.
- The Power of Constraints: No single knowledge source is perfect. However, by requiring a hypothesis to be both acoustically plausible and grammatically legal, the system can prune away thousands of incorrect interpretations.
- Approximate Knowledge as a Filter: The system uses "cheap," fast KSs to generate many rough word candidates, then applies "expensive," high-precision KSs (like the WIZARD module) to verify only the most promising ones.
- Handling Errorful Data: The system accepts that lower-level data will be "error-full." Instead of trying to fix the data perfectly at the start, it maintains multiple competing hypotheses until higher-level context can resolve the winner.
Key Results
- Accuracy: The system achieved a 90% correct semantic interpretation rate on the 1011-word task. Even when it didn't get the words 100% correct, it understood the meaning of the query 91% of the time.
- Efficiency: Opportunistic scheduling was proven essential. In experiments, an opportunistic strategy was twice as fast and significantly more accurate than a strategy that executed KSs in a fixed, non-prioritized order.
- Computational Load: The system required approximately 60 to 85 MIPSS (million instructions per second of speech) on a 36-bit PDP-10.
Limitations and Future Work
- Generality vs. Efficiency: While the blackboard is flexible, the overhead of managing it is high. The authors note that for highly constrained tasks, "compiled" systems like HARPY (which pre-computes all possible sentence paths into a massive network) are faster and more accurate.
- Weak Juncture Models: Hearsay-II's model of how words blend together at their boundaries (junctures) was identified as a weak point compared to its contemporaries.
- Internal Complexity: While the interface between KSs is simple, the internal logic of individual KSs often required private, complex data structures that didn't fit the uniform blackboard model, leading to some "specialization" that limited the system's theoretical purity.
The authors conclude that while Hearsay-II might be "overkill" for simple speech tasks, its architecture is a blueprint for any AI system that must integrate diverse, uncertain expertise to solve complex, real-world problems.
Summary updated