Abstract
This paper presents a formal methodology for evaluating the performance of expert systems by focusing on their "inference structure"—the logical arrangement of rules and knowledge—rather than just the underlying code execution. By modeling these structures using Petri Nets and applying z-transform functions, the authors provide a quantitative framework for predicting efficiency and search complexity early in the development lifecycle.
Introduction
In the early era of expert systems, performance evaluation was often treated as a "black art." Most metrics were confined to the program level, measuring execution speed or memory usage. However, the true performance of an expert system is fundamentally dictated by its "knowledge level"—the intrinsic relationships and dependencies between rules and data. A system may possess high-quality rules yet suffer from a poorly organized inference structure, leading to combinatorial explosions in search spaces. This paper addresses the need for tools that allow knowledge engineers to compare organizational strategies and predict scalability before full implementation.
Methodology
The proposed framework consists of two primary components: a network model and a performance function.
The Network Model
The authors represent the inference structure as a directed graph based on Petri Net theory. In this model, "states" represent discrete pieces of knowledge or data items, while "transitions" represent the rules that facilitate movement between these states.
Z-Transform Performance Function
To quantify the efficiency of the network, a "performance factor" ($p$) is assigned to each rule transition, representing its associated cost or complexity. From this, a z-transform function is derived: $$f(z) = \sum N(p) z^p$$ where $N(p)$ is the mean number of reasoning paths with a specific performance cost $p$. This function allows for the derivation of several key metrics, including the Breadth Factor (BF), which measures the total search space, and the Depth Factor (DF), which represents the average effort of a reasoning path.
Evaluation and Results
The methodology was validated using the Geological Prospecting Expert system (GPE), a real-world application for mineral exploration. The evaluation provided a concrete "scorecard" for GPE ($BF = 59.09, DF = 5.70, |S| = 120$), establishing a baseline for future refinements. Furthermore, the authors developed EVAL, an interactive software tool within the KMIX environment, to automate these calculations, allowing engineers to "preview" performance during the design phase.
Key Insights
The Cost of Generalization
A significant finding is the quantitative trade-off in rule generalization. By refining and generalizing rule sets, engineers can reduce the Breadth Factor (and thus the search space) by 25% to 67%. While this may slightly increase the Depth Factor, the overall efficiency gains are substantial.
Structural Optimization
The research demonstrates that the topology of the inference network exponentially impacts performance. For instance, comparing linear structures to binary tree structures for pattern matching reveals how different shapes can mitigate or exacerbate search complexity as the number of variables increases.
Conclusion and Future Work
While this framework provides a robust foundation for structural analysis, limitations remain. Assigning "performance factors" to transitions can be subjective, requiring standardized criteria across domains. Additionally, the current model focuses on static structures; future work is needed to account for dynamic control strategies and the impact of data volume on system performance. Nonetheless, shifting the focus to the knowledge level offers a more profound understanding of expert system efficiency.
Summary updated