FRI - Feature Relevance Intervals for Interpretable and Interactive Data Exploration

Lukas Pfannschmidt, Christina Göpfert, Ursula Neumann, Dominik Heider, Barbara HammerCIBCB 2019 · DOI · OpenReview

TL;DR

This paper introduces FRI, a Python library for feature selection that goes beyond identifying the "best" subset of features. FRI provides "relevance intervals" for each feature, revealing their potential contribution across all well-performing models, thus offering a more nuanced understanding of feature importance, especially in the presence of redundant or correlated features. This approach facilitates interactive data exploration and biomarker discovery.

What Problem Does This Paper Address?

In fields like biomedical data analysis, high-dimensional datasets and complex sensor signals are common. Traditional feature selection methods often struggle with these scenarios, sometimes selecting features based on spurious correlations rather than true causal relationships. Furthermore, many methods aim to find a minimal set of features for optimal prediction, which can obscure the importance of all relevant features, including those that are weakly relevant or redundant. This lack of comprehensive understanding hinders data exploration, the discovery of alternative biomarkers, and the interpretation of model behavior. The paper addresses this by proposing a method to identify "all-relevant" features and provide interpretable insights into their roles, even when multiple equivalent models exist.

Methods

The core of the proposed method is the "Feature Relevance Interval" (FRI). Instead of assigning a single coefficient or importance score to a feature, FRI determines an interval of possible weights (or relevance bounds) for that feature across the entire class of well-performing linear models. This is achieved through a multi-step process:

  1. Baseline Solution: An initial linear classifier (specifically, an L1-regularized soft-margin SVM) is trained to establish a baseline.
  2. Equivalent Classifier Class: Constraints are derived from this baseline to define a class of "equivalent" classifiers that achieve similar prediction accuracy. This class encompasses all models that perform comparably.
  3. Minimum and Maximum Bounds: For each feature, linear programming is used to find the minimum and maximum possible weights it can have across all classifiers within the equivalent class. These define the feature's relevance interval.
  4. Feature Classification: Based on these intervals, features are classified into three categories:
  5. Strongly Relevant: Lower bound is significantly greater than zero.
  6. Weakly Relevant: Lower bound is zero, but the upper bound is greater than zero. These features can substitute for each other or other strongly relevant features.
  7. Irrelevant: Both bounds are zero.
  8. Interactive Constraints: The framework allows users to impose custom constraints on feature relevance (e.g., fixing a feature's relevance to a specific value or range), enabling interactive exploration and hypothesis testing.

Datasets

The paper evaluates FRI using both simulated and real-world biomedical datasets:

  • Simulated Data: Five sets (Sim1-Sim5) were generated with varying numbers of strongly relevant, weakly relevant, and irrelevant features, totaling 30 features and 500 samples each. These datasets allow for explicit evaluation against a known ground truth.
  • Biomedical Data:
    • t21: Prenatal examination data for early diagnosis of chromosomal abnormalities (trisomy 21), with 18 features.
  • flip: Data for predicting fibrosis, with 118 patients and 19 features.
  • spectf: Cardiac Single Proton Emission Computed Tomography (SPECT) images, with 44 features.
  • wbc: Cell image-based breast cancer diagnostics, with 32 markers.
  • colposcopy: Extracted structural features from colposcopy videos, with 69 features.

The spectf, wbc, and colposcopy datasets were sourced from the UCI Machine Learning Repository.

Key Insights and Ideas

  • Beyond Minimal Feature Sets: The most significant insight is the shift from finding a single "best" minimal feature set to understanding the range of relevance for all features. This is crucial for interpretability and discovering alternative solutions.
  • Handling Redundancy: FRI explicitly addresses feature redundancy by identifying weakly relevant features. This allows for understanding functional substitutions and potentially selecting less costly or invasive biomarkers.
  • Interactive Exploration: The ability to impose constraints on feature relevance transforms feature selection from a batch process into an interactive dialogue with the data, allowing domain experts to guide the analysis.
  • Interpretable Visualizations: The relevance intervals can be visualized, providing an intuitive way to grasp feature importance and relationships, which is highly beneficial for biomedical experts.
  • Robustness to Equivalent Models: By considering the entire class of equivalent models, FRI provides more stable and reliable feature relevance estimates compared to methods relying on a single model.

Key Results

  • Simulated Data Performance: FRI achieved near-perfect F1 scores across all simulated datasets, outperforming other methods like Boruta, EFS, ElasticNet, and Stability Selection, especially in scenarios with weakly relevant features.
  • Biomedical Data Performance: While no single method dominated across all biomedical datasets in terms of ROC-AUC, FRI demonstrated strong performance, particularly excelling in the colposcopy dataset. It also provided valuable insights into feature relationships, such as identifying the association between gestational age and fetal length in the t21 dataset.
  • Feature Set Size: FRI's feature sets were comparable in size to those identifying only strongly relevant features, suggesting high precision, while also retaining the valuable information from weakly relevant features.
  • Runtime: FRI's runtime was competitive, especially considering its parallelizable nature and the complexity of its approach.

Limitations and Future Work

  • Focus on Linear Models: The current implementation and theoretical framework are primarily focused on linear classification and regression models. Extending this to non-linear models would be a significant advancement.
  • Computational Cost: While parallelizable, the feature classification step (involving permuted features) can introduce a computational hit, particularly for smaller datasets.
  • Future Work: The authors plan to integrate a function for automatically grouping features based on constrained relevance bounds. Further development could explore more sophisticated interactive exploration tools and extensions to non-linear models.

Summary updated

Abstract

Most existing feature selection methods are insufficient for analytic purposes as soon as high dimensional data or redundant sensor signals are dealt with since features can be selected due to spurious effects or correlations rather than causal effects. To support the finding of causal features in biomedical experiments, we hereby present FRI, an open source Python library that can be used to identify all-relevant variables in linear classification and (ordinal) regression problems. Using the recently proposed feature relevance method, FRI is able to provide the base for further general experimentation or in specific can facilitate the search for alternative biomarkers. It can be used in an interactive context, by providing model manipulation and visualization methods, or in a batch process as a filter method.
BibTeX
@inproceedings{pfannschmidt2019fri,
  author = {Lukas Pfannschmidt and Christina Göpfert and Ursula Neumann and Dominik Heider and Barbara Hammer},
  title = {{FRI - Feature Relevance Intervals for Interpretable and Interactive Data Exploration}},
  booktitle = {CIBCB},
  year = {2019},
  doi = {10.1109/cibcb.2019.8791489},
}
Figure