TL;DR
This paper introduces a computer-supported methodology for comparing the conceptual systems of multiple experts to identify and resolve disagreements in knowledge acquisition for expert systems. By analyzing how experts define concepts and distinctions, the method makes tacit disagreements overt, facilitating more robust and reliable knowledge-based systems.
What Problem Does This Paper Address?
Developing knowledge-based systems often requires acquiring knowledge from multiple experts, as expertise is rarely concentrated in a single individual. A significant challenge arises when these experts disagree on the use of concepts and vocabulary, leading to confusion and potential errors in the resulting system. This disagreement can be subtle and tacit, making it difficult to identify and address. The paper tackles the problem of how to systematically elicit, compare, and visualize these conceptual differences among experts to build more accurate and reliable knowledge-based systems.
Methods
The core of the methodology relies on entity-attribute grid elicitation. This technique involves asking experts to:
- Identify a set of entities relevant to a specific domain.
- Make distinctions among these entities.
- Name these distinctions using specific terminology.
- Classify each entity based on these distinctions.
The paper proposes a computer-supported approach to analyze these grids. Key aspects include:
- Comparing conceptual systems: The methodology compares the distinctions and terminology used by different experts.
- Measuring similarity: A fuzzy set model is used to quantify the similarity between distinctions made by different experts. The formula
Ve (x - y) = ^ (xe = ye)is introduced to measure equivalence between two attributes (distinctions). - Classifying relationships: Based on the similarity measures, relationships between experts' conceptual systems are classified into four categories:
- Consensus: Experts use the same term for the same distinction.
- Conflict: Experts use the same term for different distinctions.
- Correspondence: Experts use different terms for the same distinction.
- Contrast: Experts use different terms for different distinctions.
- Graphical presentation: The results are visualized using graphical representations (Figures 5, 7) to make areas of potential dissent readily apparent to the experts, facilitating discussion and resolution.
The methodology is structured into three phases:
- Domain Discussion and Instantiation: Experts agree on a common set of entities and the purpose of the elicitation.
- Conceptualization and Feedback: Experts individually create grids based on the common entities.
- Exchange and Compare: Experts classify entities using each other's attributes, and the resulting grids are analyzed for consensual and conflicting attributes.
Datasets
Not discussed in the paper. The paper focuses on a methodology and uses examples from geographic mapping techniques and general expert system domains for illustration, rather than specific, named datasets.
Key Insights and Ideas
- Tacit disagreement is a major hurdle: The paper highlights that many disagreements in knowledge acquisition are not explicit but are embedded in how experts use language and concepts.
- Visualizing differences is crucial: Presenting the comparison results graphically (e.g., using quadrant diagrams and graphs showing match values) is essential for experts to understand and discuss their differences effectively.
- Distinguishing terminology vs. concept differences: The methodology clearly differentiates between "conflict" (same term, different concept) and "correspondence" (different term, same concept), which are critical for understanding the nature of expert disagreement.
- Methodology scales: The approach is designed to scale from pairwise comparisons to group comparisons, making it applicable to real-world scenarios with multiple experts.
- Objective analysis: The computer-driven analysis avoids subjective opinions about whose usage is "right" or "wrong," allowing experts to focus on understanding and resolving differences.
Key Results
The paper demonstrates the application of its methodology through illustrative examples and figures.
- Figure 5 shows a consensus and conflict analysis for two experts in geographic mapping, visually presenting areas of agreement and disagreement.
- Figure 7 summarizes comparisons between two experts into a four-quadrant diagram, categorizing attribute relationships as Consensus (>80), Correspondence (>80), Conflict (<70), and Contrast (<70).
- The methodology provides a structured way to identify specific points of contention, such as the use of "linear interpolation" or the entity "probability matching" in the geographic mapping example.
Limitations and Future Work
- Approximation: The authors acknowledge that comparing conceptual systems involves approximation, as complete conceptual systems can be indefinitely complex.
- Focus on extensionally defined distinctions: The method primarily works with distinctions as they are applied to a set of entities, which is an extensional approach and may miss deeper conceptual nuances.
- Resolution of conflict: While the methodology makes conflict apparent, the paper states that the computer-based system can go "no further than making the basis of conflict and the possible need for resolution apparent." The actual resolution process remains a human-driven discussion.
- Future work: The paper suggests that the insights gained from comparing conceptual systems can be used to inform further knowledge acquisition steps, such as defining classes, properties, and rules, and generating alternative explanations or questions. The interpretation of correspondences can lead to significant insights into abstraction levels and knowledge structure representation.
Summary updated