A Critical View on Automatic Significance-Filtering in Pattern Mining

Florian Lemmerich, Frank PuppeSSDM@ECML/PKDD 2014 · OpenReview

TL;DR

This paper argues against the mandatory automatic filtering of data mining results based solely on statistical significance. It proposes that while statistical validation is important, forcing all discovered patterns through a significance test without considering the application context can hinder valuable exploratory analysis and lead to the rejection of potentially useful insights.

What Problem Does This Paper Address?

In data mining, a key goal is to discover "valid, novel, potentially useful and ultimately understandable patterns." A crucial aspect of "validity" is ensuring patterns aren't just random noise. Many approaches use statistical significance tests to filter out patterns that don't meet a certain threshold. This is often done automatically, without human intervention, to ensure statistical soundness. However, this paper questions whether this automatic, mandatory filtering is always the best approach. It highlights that inexperienced users might blindly reject patterns that don't pass a significance test, potentially missing valuable leads. The paper aims to encourage a more critical assessment of automatic statistical filtering, suggesting it shouldn't be a one-size-fits-all mandatory step.

Methods

The paper doesn't introduce new algorithms or techniques. Instead, it presents a critical analysis of existing practices in automatic significance filtering for pattern mining. It structures its arguments around four main propositions:

  1. Data Mining as Hypothesis Generation, Not Testing: The paper argues that data mining's primary strength lies in exploratory tasks and generating hypotheses, rather than definitively testing them.
  2. Independent Significance Testing: It suggests that performing significance testing as a separate step, after initial pattern discovery and human inspection, can lead to better results. This allows for expert selection of patterns and optimization of statistical tests.
  3. Limitations of Interactive Mining: Current significance testing adaptations for multiple comparisons are not well-suited for interactive and iterative data mining processes, as they can lead to counter-intuitive results and discourage free exploration.
  4. Significance vs. Interestingness: The paper contends that statistical significance alone doesn't equate to "interestingness." Significance tests are heavily influenced by dataset size, potentially filtering out important but rare patterns or highlighting common but weak ones.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • The "Exploratory" Nature of Data Mining: The paper strongly emphasizes that data mining is fundamentally an exploratory process. Forcing strict statistical validation at every step can stifle this exploration and prevent the discovery of unexpected but valuable insights.
  • Human Expertise is Crucial: While automatic methods are convenient, human experts can identify semantically redundant patterns, select potentially useful ones, and optimize the choice of statistical tests. This manual inspection can lead to more powerful and relevant findings.
  • Interactive Mining Challenges: Current methods for handling multiple comparisons in significance testing are not designed for the iterative and interactive nature of modern data mining. This can make the process cumbersome and less effective.
  • Significance Isn't Everything: The paper argues that a statistically significant pattern isn't necessarily the most interesting or useful one. In some applications, a pattern that applies to a small but specific group might be far more valuable than one that shows a marginal difference across a large population.

Key Results

The paper doesn't present quantitative results from experiments. Its contribution is qualitative and argumentative, aiming to shift the perspective on automatic significance filtering. The "results" are the logical conclusions drawn from its propositions, advocating for a more nuanced approach to statistical validation in pattern mining.

Limitations and Future Work

The authors acknowledge that their arguments are presented as a position paper and are not exhaustive. They suggest that a trade-off is often necessary when deciding on automatic filtering, and the decision should be made based on the specific problem at hand.

Future work could involve developing better methods for statistically sound, interactive, and iterative pattern mining. The paper also implies a need for more research into how to balance statistical significance with other measures of pattern interestingness, especially in contexts where dataset size heavily influences significance. The authors also suggest that the decision factors for automatic filtering (listed in Section 3) could guide future research and system design.

Summary updated

Abstract

Statistically sound validation of results plays an important role in modern data mining. In this context, it has been advocated to disregard patterns that cannot be automatically confirmed as stati...
BibTeX
@inproceedings{lemmerich2014critical,
  author = {Florian Lemmerich and Frank Puppe},
  title = {{A Critical View on Automatic Significance-Filtering in Pattern Mining}},
  booktitle = {SSDM@ECML/PKDD},
  year = {2014},
  url = {https://openreview.net/forum?id=XNGB0-cPxi},
}
Figure