Introduction
In the mid-1980s, the field of artificial intelligence faced a significant challenge: the "brittleness" of expert systems. While these systems performed admirably in narrow, well-defined domains, they often failed catastrophically when encountering scenarios slightly outside their programmed expertise. This failure was widely attributed to "shallow" modeling—relying on surface-level heuristics rather than an underlying understanding of the domain. However, the distinction between "deep" and "shallow" remained largely intuitive.
This 1987 paper from the University of Pennsylvania provides a formal framework to characterize "knowledge depth," specifically within the context of Intelligent Safety Systems (ISS). These systems, designed for high-stakes environments like nuclear reactors and chemical plants, require a rigorous approach to modeling to ensure robustness and explainability.
The "Deeper-Than" Relation
The authors propose an operational definition to move beyond vague terminology. They define a "deeper-than" relation: Model M' is deeper than model M if there exists some implicit knowledge in M which is explicitly represented or computed in M'.
By this definition, depth is not an absolute metric but a comparative one. Increasing depth involves taking logic that was previously "baked into" the code and making it an explicit part of the system's reasoning process. This allows the system to reason about why a certain action is appropriate, rather than simply following a pre-set instruction.
The Spectrum of Plan Determination
The paper decomposes the task of providing a safety function into two primary components: Plan Determination and Plan Evaluation. For Plan Determination—the process of identifying actions to circumvent a crisis—the authors identify four distinct levels of increasing depth:
- Hard-coded Plans: This is the shallowest level, characterized by simple "if-then" rules. The system's understanding of the underlying structure is entirely implicit.
- Hard-coded Paths: Systems at this level use "response trees" that explicitly list valid paths between components. While the paths are explicit, the physical logic of why these paths work remains implicit.
- System Structure: Here, the system represents the environment as a graph. Plans are not pre-written but are generated dynamically via graph search algorithms.
- Structure + Behavior: This is the deepest level, where the system reasons about the physical behavior of individual components.
At this deepest level, the system can simulate the effects of a specific failure—such as a stuck valve in a reactor—and determine the consequences based on physical principles rather than pre-defined failure scenarios.
Plan Evaluation and Utility
Once potential plans are identified, the system must select the best alternative. Depth in Plan Evaluation moves from arbitrary numeric priorities to Multi-attribute Utility Theory (MAUT). In shallow models, priorities are assigned by human experts. In deep models, the utility scores (e.g., the cost of an action or the speed of recovery) are computed directly from the structural and behavioral model of the system, ensuring that the evaluation is grounded in the current state of the physical environment.
The Knowledge Engineering Trade-off
While deeper models offer significant advantages in portability and robustness, they are not a universal solution. The authors highlight a "knowledge engineering tax": deeper models are more complex to build and require more computational power to execute. In stable environments where failure modes are well-understood, a shallow, efficient heuristic model may be more practical than a deep, slow reasoning engine. The goal for developers is to find the "optimal" depth that balances safety requirements with computational constraints.
Conclusion
By formalizing the concept of knowledge depth, this paper provided a roadmap for building more resilient AI systems. The transition from implicit heuristics to explicit behavioral modeling remains a cornerstone of modern safety-critical AI, ensuring that systems can handle the unexpected with more than just a set of hard-coded rules.
Summary updated