Post-Analysis of Learned Rules

Bing Liu, Wynne Hsu

In the mid-1990s, machine learning algorithms like C4.5 were highly effective at inducing rules from large datasets. However, these systems often presented a significant challenge: they generated hundreds or thousands of rules, many of which were either common sense or irrelevant to the user's specific goals. This classic paper from AAAI-96 by Bing Liu and Wynne Hsu addresses this "now what?" phase of data mining, introducing a fuzzy logic framework to help humans reconcile automated insights with their own intuition.

The Problem of "Interestingness"

The authors identify a critical gap in rule induction research. While objective measures like accuracy and coverage are useful, they operate in a vacuum. In real-world applications, "interestingness" is inherently subjective. Users are not blank slates; they possess prior knowledge and expectations. A rule is truly interesting only if it either confirms a hypothesis, contradicts a long-held belief, or provides a novel extension to existing knowledge.

The Fuzzy Matching Framework

To bridge the gap between machine-learned precision and human intuition, the paper proposes a two-step post-analysis process.

1. Fuzzy Specification

Users express their prior knowledge or hypotheses as fuzzy rules. Instead of using rigid boundaries (e.g., Age < 30), they employ linguistic variables (e.g., Age = young). This is achieved through membership functions, typically represented as trapezoidal curves.

Figure 2: A trapezoidal membership function used to define linguistic variables like 'young' or 'fast'.

As shown in Figure 2, a membership function allows a value to have a degree of membership between 0 and 1. For instance, a value between $b$ and $c$ might be "definitely" part of the set, while values between $a$ and $b$ or $c$ and $d$ represent the "fuzzy" boundaries of human concepts.

2. Matching and Ranking

Once the user's expectations are defined, the system matches newly induced rules against this set of existing knowledge ($E$). It calculates a match weight based on attribute names, values, and operators.

Figure 1: The matching process between induced rules (B) and the user's existing knowledge base (E).

Figure 1 illustrates how each induced rule $B_i$ is compared against the entire set of user-defined rules $E$. The final weight $W_i$ for a rule is determined by the maximum match found across the existing knowledge base, allowing the system to rank rules by their relevance to the user's mental model.

Categorizing Insights

The system doesn't just provide a single similarity score; it categorizes rules into three distinct types of "interestingness":

  • Rule-Similarity: The induced rule confirms the user's intuition, providing empirical support for existing beliefs.
  • Unexpected Consequent: The conditions of the rule match the user's expectation, but the resulting outcome is different. These are often the most valuable "aha!" moments.
  • Unexpected Conditions: The outcome is what the user expected, but the underlying reasons or conditions are entirely different or even contradictory to their prior assumptions.

Evaluation and Impact

The authors demonstrated the system's effectiveness using an accident database and an encoded confidential dataset. The system successfully ranked rules that confirmed expectations at the top when searching for similarity, and flagged critical anomalies (like unexpected injury levels for certain driving conditions) when searching for differences.

One of the paper's most enduring insights is that "different" is not simply the inverse of "similar." Identifying contradictions requires specific algorithmic logic distinct from similarity matching. By using fuzzy logic as a "Human-AI Bridge," Liu and Hsu created a framework where subjectivity is treated as a feature rather than a bug, allowing for a more interactive and insightful data mining process.

Conclusion

While modern machine learning has moved toward deep learning and black-box models, the problem of interpretability and "interestingness" remains as relevant as ever. This 1996 paper reminds us that the ultimate goal of data mining is not just to find patterns, but to provide actionable knowledge that resonates with—or meaningfully challenges—human understanding.

Summary updated

Abstract

Rule induction research implicitly assumes that after producing the rules from a dataset, these rules will be used directly by an expert system or a human user. In real-life applications, the situation may not be as simple as that, particularly, when the user of the rules is a human being. The human user almost always has some previous concepts or knowledge about the domain represented by the dataset. Naturally, he/she wishes to know how the new rules compare with his/her existing knowledge. In dynamic domains where the rules may change over time, it is important to know what the changes are. These aspects of research have largely been ignored in the past. With the increasing use of machine leaming tcclmiques in practical applications such as data mining, this issue of post analysis of rules warrants greater emphasis and attention. In this paper, we propose a technique to deal with this problem. A system has been implemented to perform the post analysis of classification rules genemted by systems such as C4.5. The proposed technique is general and highly interactive. It will be particularly useful in data mining and data analysis.
BibTeX
@inproceedings{liu1996post,
  author = {Bing Liu and Wynne Hsu},
  title = {{Post-Analysis of Learned Rules}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}
Figure