Abstract: Expert systems often rely on shallow empirical rules that lack deep domain understanding, making them difficult to debug when they fail. This paper presents a framework for Knowledge Base Refinement using explicit justification structures—dependency networks that link empirical rules to underlying domain principles. By categorizing error types and tracing them through these networks, a Learning Apprentice System (LAS) can automatically isolate faulty assumptions and suggest repairs.
Introduction
In the early days of expert systems research, a significant challenge was the "knowledge acquisition bottleneck." Systems relied on "shallow" rules—empirical associations that captured expert heuristics but lacked a deep understanding of the underlying physics or logic. When these systems failed, identifying the root cause was difficult. Previous tools like TEIRESIAS used reasoning traces to show which rules fired, but they could not explain the logic behind the rules themselves. This paper proposes a Learning Apprentice System (LAS) that uses explicit "justification structures" to bridge the gap between shallow rules and deep domain theory.
Justification Structures and Error Taxonomy
The core of the LAS framework is the justification structure: a dependency network where each "Belief" (a rule or assertion) is linked to the "Justifier Beliefs" that support it. These structures record the derivation of a rule from first principles, including the assumptions and approximations made along the way. Linkage rules define these connections as either deductive (logical proofs) or abductive (interpretive leaps).
To facilitate automated debugging, the authors define a specific taxonomy of errors that can occur within these structures.
This taxonomy allows the system to classify failures into Rule Belief Errors (over-general or over-specific conditions), Numeric Parameter Errors (threshold issues), or Symbolic Parameter Errors (logical comparator issues).
The Learning Apprentice System (LAS)
When a user corrects the system, a component called the "Critic" performs dependency-directed backtracking. It uses the error type and the justification structure to prune the search space. The Critic ignores "definitional" beliefs, which are assumed to be true, and focuses on "default" or "theoretical" assumptions that are more likely to be faulty. This process, known as "blame assignment," allows the system to maintain "convenient falsehoods"—useful approximations that work most of the time—while knowing exactly which "shorthand" to expand when a failure occurs.
Case Study: The Dipmeter Advisor
The primary testbed for this research is the Dipmeter Advisor, an expert system used in oil exploration to infer geological structures from borehole data. The authors analyzed the NFR12 (Late Fault Rule), a complex rule supported by a justification structure containing 65 distinct beliefs.
By tracing failures through this 65-node network, the LAS could isolate specific culprits, such as an underestimate of distortion region thickness or a fault orientation angle. The system then ranks these hypotheses based on the "Degree of Belief" and suggests specific repairs to the rule's numeric thresholds.
Conclusion
The use of explicit justifications represents a significant step toward self-refining AI systems. While the manual creation of these structures remains labor-intensive—a new form of the acquisition bottleneck—the ability to trace empirical failures back to deep domain principles provides a robust path for automated knowledge base refinement. Future work aims to automate the generation of these justifications from a general domain theory and incorporate pragmatic constraints like the utility of specific rules.
Summary updated