TL;DR
This paper investigates the challenges of maintaining XCON, a large-scale expert system for computer configuration, and introduces XCON-in-RIME, a successor system built with the RIME language. The authors argue that XCON-in-RIME's higher-order language and structured programming environment significantly improve maintainability by promoting homogeneity and predictability in the rule base, as supported by anecdotal evidence from rule developers.
What Problem Does This Paper Address?
Expert systems, like XCON, are powerful tools for automating complex decision-making tasks. However, as these systems grow in size and complexity, maintaining them becomes a significant challenge. XCON, initially a small system, has ballooned to over 6,200 rules, with half of them changing annually. This rapid evolution, while maintaining satisfactory performance, has made XCON increasingly difficult to modify and update. The core problem is the "degradation in integrity" of the rule base, where implicit knowledge, idiosyncratic coding practices, and a lack of explicit structure lead to a "rat's nest" of tightly coupled and hard-to-understand rules. This makes it difficult for developers to understand the existing code, introduce changes without unintended side effects, and preserve the overall coherence of the system.
Methods
The paper presents an assessment of XCON-in-RIME's maintainability through two main arguments:
- In Principle Argument: This argument focuses on the design of RIME, the language used for XCON-in-RIME. RIME is presented as a "higher-order language" compared to OPS5 (the language XCON was written in). This is because RIME incorporates explicit constructs for domain knowledge and problem-solving strategies, such as "problem spaces" (grouping rules by function), "problem-solving methods" (structured sequences of steps), and "rule types" (templates for rule creation). These features aim to make the implicit procedural relationships between rules explicit, thereby promoting homogeneity and predictability.
- In Practice Argument: This argument draws on anecdotal evidence from interviews with rule developers who rotated into the XCON-in-RIME project. The interviews focused on their experiences with homogeneity (how similar rules are across different developers) and predictability (knowing where to look for information and how the system behaves).
Datasets
Not discussed in the paper. The paper focuses on the XCON and XCON-in-RIME systems themselves, not on external datasets for training or evaluation.
Key Insights and Ideas
- The "Degradation in Integrity" Problem: Large, evolving rule bases tend to become less coherent and harder to understand over time, even if their performance remains acceptable. This is a fundamental challenge in maintaining complex software systems.
- RIME as a "Higher-Order Language": By providing explicit constructs for domain knowledge and problem-solving strategies, RIME moves beyond a purely domain-independent language (like OPS5) towards one that better reflects the problem domain. This is analogous to how FORTRAN is a higher-order language than assembly language.
- Homogeneity and Predictability are Key to Maintainability: The paper argues that these two properties are crucial for making code easier to understand and modify. Homogeneity means rules are written in a consistent style, while predictability means developers can anticipate where to find information and how changes will affect the system.
- Explicit Constructs Facilitate Homogeneity: Features like "problem spaces" and "subgroup schemas" in RIME provide organizational units that guide developers, leading to more consistent rule writing styles and thus greater homogeneity.
- Structured Problem-Solving Methods Enhance Predictability: The explicit steps within problem-solving methods (PROPOSE, ELIMINATE, APPLY, EVALUATE) make the flow of control more apparent, improving predictability.
- Anecdotal Evidence Supports the Design: Rule developers reported that XCON-in-RIME rules felt more homogeneous and that they could more easily pinpoint where to make changes, supporting the "in principle" arguments for RIME's design.
Key Results
- Qualitative Improvement in Maintainability: The primary result is the qualitative assessment that XCON-in-RIME, due to its RIME language and structured environment, is expected to be significantly more maintainable than XCON.
- Developer Feedback: Rotaters reported that they "couldn't tell who wrote the rules" in XCON-in-RIME, indicating a high degree of homogeneity. They also felt they "knew where to go in the rule base to add some new rule" and that "the rules are more organized," indicating improved predictability.
- Elimination of Idiosyncratic "Tricks": Developers no longer need to rely on special "tricks" to control rule firing, as RIME provides explicit mechanisms for this.
Limitations and Future Work
- Anecdotal Evidence: The "in practice" argument relies on anecdotal evidence from a small number of rotaters (initially 4, then an additional 4). The authors acknowledge that this is not "statistical evidence" and that a methodologically rigorous, controlled study would be too costly and its benefits unclear at this stage.
- Defining Metrics: The authors suggest that to gather more concrete data on maintainability, specific metrics need to be defined, such as quantifying homogeneity or measuring human performance in tasks like adding/changing rules and fixing bugs.
- Scalability: The paper raises the question of what happens when the rule base grows even larger (e.g., 18,000 or 27,000 rules), implying that current solutions might not scale indefinitely.
- Future Work: The authors advocate for developing a range of evaluation tools for expert systems as they become more of an "engineering enterprise." They also suggest that further data collection on performance and readability degradation over time would be valuable.
Summary updated