Abstract
Hierarchical data structures often contain significant vertical redundancy, making it difficult to extract truly informative patterns. This paper introduces MICA-Miner, an algorithm designed to identify "contrastive antichains"—subsets of a hierarchy that best highlight how a specific target concept differs from a reference norm. By utilizing information theory and a "subjective interestingness" measure, the method identifies non-redundant patterns that are both surprising to the user and easy to communicate.
Problem Definition
Data is frequently organized into hierarchies (e.g., Foursquare venue categories or product taxonomies). However, these levels are often redundant: if a city has an unusually high number of restaurants, it is not surprising that it also has many Italian restaurants. The challenge lies in finding the "sweet spot" in the hierarchy—the specific level of granularity that provides the most insight without being repetitive.
The authors aim to find antichains, which are sets of items where no item is an ancestor of another. These antichains must be contrastive, meaning they stand out compared to a reference or average background model.
Methodology
The researchers leverage the FORSIED framework, defining the "Subjective Interestingness" ($SI$) of a pattern as the ratio between its Information Content ($IC$) and its Description Length ($DL$).
- Background Model: To quantify "surprise," a probabilistic model of user prior beliefs is constructed using a Maximum Entropy approach. The root follows a geometric distribution, while child nodes follow binomial distributions relative to their parents.
- Information Content ($IC$): This measures the deviation of observed data from the background model.
- Description Length ($DL$): This penalizes complexity, preferring patterns closer to the root or with fewer items for easier interpretability.
- Iterative Mining & Belief Updating: A key innovation is the iterative update of the background model using the sum-product algorithm. Once a pattern is presented, the model is updated so that subsequent results are not redundant variations of previously discovered insights.
- MICA-Miner Algorithm: Finding the optimal antichain is NP-hard. The authors propose a greedy heuristic algorithm that scales quadratically with the number of items in the hierarchy.
Experimental Evaluation
The method was evaluated on three real-world datasets:
- Foursquare (FV): Comparing venue distributions in Amsterdam and London against a 20-city norm.
- Open Food Facts (OFF): Comparing food product categories between France and the USA.
- European Parliament (EP_Abst): Analyzing abstention patterns across political groups and policy topics.
Key Results and Insights
MICA-Miner demonstrated a quadratic time complexity ($O(|E|^2)$), making it practical for large hierarchies. Quantitative tests confirmed that iterative updating significantly reduces redundancy in the top results.
Qualitatively, the algorithm successfully identified unique cultural characteristics:
- In Amsterdam, it highlighted "Marijuana Dispensaries" and "Tram Stations" as over-represented.
- In French Food, it identified a high prevalence of "fresh meals" and a lack of "popcorn" relative to American data.
Conclusion
By formalizing "interestingness" as the gap between reality and expectation, and by forcing results to be antichains, MICA-Miner effectively navigates the trade-off between specificity and redundancy. Future work may involve integrating this approach into broader subgroup discovery and exceptional model mining tasks.
Summary updated