What's in a Deep Model? A Characterization of Knowledge Depth in Intelligent Safety Systems

David Klein, Timothy W. Finin

Introduction

In the mid-1980s, the field of artificial intelligence faced a significant challenge: the "brittleness" of expert systems. While these systems performed admirably in narrow, well-defined domains, they often failed catastrophically when encountering scenarios slightly outside their programmed expertise. This failure was widely attributed to "shallow" modeling—relying on surface-level heuristics rather than an underlying understanding of the domain. However, the distinction between "deep" and "shallow" remained largely intuitive.

This 1987 paper from the University of Pennsylvania provides a formal framework to characterize "knowledge depth," specifically within the context of Intelligent Safety Systems (ISS). These systems, designed for high-stakes environments like nuclear reactors and chemical plants, require a rigorous approach to modeling to ensure robustness and explainability.

The "Deeper-Than" Relation

The authors propose an operational definition to move beyond vague terminology. They define a "deeper-than" relation: Model M' is deeper than model M if there exists some implicit knowledge in M which is explicitly represented or computed in M'.

By this definition, depth is not an absolute metric but a comparative one. Increasing depth involves taking logic that was previously "baked into" the code and making it an explicit part of the system's reasoning process. This allows the system to reason about why a certain action is appropriate, rather than simply following a pre-set instruction.

The Spectrum of Plan Determination

The paper decomposes the task of providing a safety function into two primary components: Plan Determination and Plan Evaluation. For Plan Determination—the process of identifying actions to circumvent a crisis—the authors identify four distinct levels of increasing depth:

  1. Hard-coded Plans: This is the shallowest level, characterized by simple "if-then" rules. The system's understanding of the underlying structure is entirely implicit.
Example of a shallow, hard-coded rule for job management.
  1. Hard-coded Paths: Systems at this level use "response trees" that explicitly list valid paths between components. While the paths are explicit, the physical logic of why these paths work remains implicit.
  2. System Structure: Here, the system represents the environment as a graph. Plans are not pre-written but are generated dynamically via graph search algorithms.
  3. Structure + Behavior: This is the deepest level, where the system reasons about the physical behavior of individual components.
A structural and behavioral model of a reactor system, representing a deeper level of knowledge.

At this deepest level, the system can simulate the effects of a specific failure—such as a stuck valve in a reactor—and determine the consequences based on physical principles rather than pre-defined failure scenarios.

Plan Evaluation and Utility

Once potential plans are identified, the system must select the best alternative. Depth in Plan Evaluation moves from arbitrary numeric priorities to Multi-attribute Utility Theory (MAUT). In shallow models, priorities are assigned by human experts. In deep models, the utility scores (e.g., the cost of an action or the speed of recovery) are computed directly from the structural and behavioral model of the system, ensuring that the evaluation is grounded in the current state of the physical environment.

The Knowledge Engineering Trade-off

While deeper models offer significant advantages in portability and robustness, they are not a universal solution. The authors highlight a "knowledge engineering tax": deeper models are more complex to build and require more computational power to execute. In stable environments where failure modes are well-understood, a shallow, efficient heuristic model may be more practical than a deep, slow reasoning engine. The goal for developers is to find the "optimal" depth that balances safety requirements with computational constraints.

Conclusion

By formalizing the concept of knowledge depth, this paper provided a roadmap for building more resilient AI systems. The transition from implicit heuristics to explicit behavioral modeling remains a cornerstone of modern safety-critical AI, ensuring that systems can handle the unexpected with more than just a set of hard-coded rules.

Summary updated

BibTeX
@inproceedings{klein1987what,
  author = {David Klein and Timothy W. Finin},
  title = {{What's in a Deep Model? A Characterization of Knowledge Depth in Intelligent Safety Systems}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1987},
}
Figure