Book Review: Computational Methods of Feature Selection

Longbing Cao, David TaniarIEEE Intell. Informatics Bull 2008 · OpenReview

Abstract

In the modern landscape of machine learning and data science, the "curse of dimensionality" presents a persistent challenge. Computational Methods of Feature Selection, edited by Huan Liu and Hiroshi Motoda, provides a comprehensive roadmap for navigating this complexity. This review explores the book's multi-disciplinary approach to identifying informative variables within high-dimensional datasets, spanning foundational theories to domain-specific applications in bioinformatics and text mining.

Introduction: The Challenge of High-Dimensional Data

As we move deeper into the era of Big Data, researchers and practitioners are frequently confronted with datasets where the number of features far exceeds the number of observations. This phenomenon, often referred to as the "curse of dimensionality," leads to several critical issues: overfitting, prohibitive computational costs, and a significant loss of model interpretability.

Feature selection—the process of identifying a subset of relevant features for use in model construction—is the primary defense against these challenges. The book edited by Liu and Motoda addresses the gap between traditional filter/wrapper methods and the sophisticated techniques required for today's heterogeneous data environments, which often include a mix of labeled, unlabeled, and partially labeled information.

The cover of "Computational Methods of Feature Selection," edited by Huan Liu and Hiroshi Motoda.

Methodological Framework

The book is structured into four major sections, each addressing a different facet of the feature selection problem:

1. Foundations and Randomization

Beyond standard deterministic approaches, the text explores stochastic methods such as Las Vegas and Monte Carlo algorithms. These are essential for managing the inherent randomness in large-scale feature selection. Furthermore, it introduces Causal Bayesian networks, moving the conversation from mere statistical correlation to causal discovery. By identifying the "Markov blanket" of a variable, researchers can pinpoint the underlying mechanisms driving the data.

2. Active and Incremental Learning

In scenarios where data acquisition is expensive or data arrives in a continuous stream, traditional batch processing fails. The book discusses active feature value acquisition, where models strategically determine which feature values are worth the cost of measurement. For massive, streaming datasets, it covers incremental ranked usefulness, allowing for real-time processing without the need to store the entire dataset.

3. Local vs. Global Weighting

A significant contribution of the text is its deep dive into the Relief family of algorithms. These methods use local distance metrics to estimate feature relevance, providing a more nuanced view than global metrics. The book includes a mathematical proof demonstrating that these algorithms are equivalent to solving an online convex optimization problem, grounding these heuristic-based methods in rigorous theory.

4. Domain-Specific Applications

The final section applies these theories to specialized fields:

  • Text Mining: Exploration of N-grams and posterior inclusion probability.
  • Bioinformatics: Focus on ensemble methods for stable biomarker discovery and penalty-based selection (e.g., Lasso) to handle genomic data where features vastly outnumber samples.

Key Insights and Theoretical Shifts

Several transformative ideas emerge from this collection:

  • Causality over Correlation: Shifting the focus to causal relationships builds models that are more robust to shifts in data distribution, as they capture the "why" rather than just the "what."
  • The Power of Sparsity: In domains like text mining, the book demonstrates that aggressive selection can remove over 95% of features while maintaining or improving model accuracy.
  • Local Relevance: Global importance can mask critical local patterns. Adaptive metrics looking at local neighborhoods can uncover relevance that broader methods overlook.
  • Ensembles for Stability: To address the instability of feature selection in sensitive fields like medicine, the book advocates for ensemble-based selection to find a "consensus" of features, ensuring reproducibility.

Quantitative Results and Performance

The review highlights several key performance metrics discussed in the book:

  • Active Sampling: This approach significantly outperforms random sampling by utilizing mixture models for joint class-feature distributions.
  • Decision Boundaries: The Bayes risk weighted vector quantization (BVQ) algorithm produces decision boundaries comparable to Support Vector Machines (SVMs) and superior to traditional multi-layer perceptrons.
  • Efficiency: The aforementioned 95% reduction in feature space for text mining serves as a benchmark for the efficiency gains possible through these methods.

Conclusion and Future Directions

While Computational Methods of Feature Selection provides state-of-the-art guidance, the field continues to evolve. The reviewers, Longbing Cao and David Taniar, note that the "small $n$, large $p$" problem in bioinformatics remains a significant hurdle. Future research must focus on better integrating domain knowledge into the selection process and developing even more scalable algorithms for the next generation of massive, heterogeneous data sources.

Reviewers: Longbing Cao (University of Technology Sydney) and David Taniar (Monash University). Source: IEEE Intelligent Informatics Bulletin, November 2008.

Summary updated

BibTeX
@inproceedings{cao2008book,
  author = {Longbing Cao and David Taniar},
  title = {{Book Review: Computational Methods of Feature Selection}},
  booktitle = {IEEE Intell. Informatics Bull},
  year = {2008},
  url = {https://openreview.net/forum?id=ljH_Uz0cN9x},
}
Figure