TL;DR
This paper introduces "Context-Aware Feature Selection and Classification" (CFSC), a novel joint model that mimics human decision-making by selecting relevant features for each specific instance before making a classification. This approach leads to more interpretable and accurate predictions, outperforming existing methods on combined classification and feature selection metrics.
What Problem Does This Paper Address?
In many decision-support systems, machine learning models are often treated as "black boxes," making it difficult for humans to understand why a particular decision was made. While interpretability is crucial, traditional methods like global feature selection (using the same features for all instances) or simple decision trees can still be overwhelming or not truly reflective of how humans make decisions. Humans naturally focus on a subset of relevant information for each specific situation. This paper addresses the gap by proposing a model that can perform feature selection contextually, meaning the selected features are specific to each individual data point, just like a human expert would. This aims to bridge the gap between high accuracy and genuine human-understandable explanations.
Methods
The core of the paper is the Context-Aware Feature Selection and Classification (CFSC) model. It's a neural network architecture designed to perform two tasks simultaneously:
- Context-Aware Feature Selection Module: This module takes an input instance and outputs a "feature mask" indicating which features are relevant for that specific instance. This is achieved by training a part of the network to predict instance-level feature labels (provided by simulated experts or human annotations).
- Classification Module: This module then uses only the features selected by the feature selection module (via the generated mask) to make the final classification decision.
The model is trained jointly, optimizing for both accurate classification and accurate feature selection. The loss function is a weighted combination of classification loss and feature selection loss, allowing for control over the trade-off between the two. A key aspect is the use of a ReLU activation on the feature selection module's output, effectively acting as a mask that zeroes out non-selected features.
Datasets
The authors evaluated their CFSC model on a combination of real-world and synthetic datasets:
- Real-world datasets: Credit, Company, Mobile, NHIS, and Ride. These datasets were chosen for their relative ease of understanding for laypeople and varied in their number of features. Since these datasets lacked instance-level feature labels, the authors created "simulated experts" to generate these labels.
- Synthetic datasets: Three datasets (Synthetic1, Synthetic2, Synthetic3) were created with instance-level feature labels. These were generated by first training a decision tree on the data and then using the decision paths of the tree to define which features were "used" for classification. This allowed for controlled experiments with known ground truth for feature selection.
Key Insights and Ideas
- Human-like decision-making is key: The paper strongly advocates for models that mimic how humans process information – by focusing on relevant details for each situation rather than applying a one-size-fits-all approach.
- Joint learning is superior: Simultaneously learning to select features and classify leads to better performance than pipeline approaches or models that only focus on one aspect.
- Instance-level feature labels are valuable: While costly to obtain, these labels are crucial for training models that can provide truly context-aware explanations.
- Simulated experts are a viable proxy: The paper demonstrates that well-designed simulated experts (using evidence counterfactuals or decision trees) can generate realistic instance-level feature labels for training and evaluation when human annotations are unavailable.
Key Results
- Outperformance on combined metrics: CFSC significantly outperformed all baselines on a combined measure of classification and feature selection performance across multiple datasets and simulation strategies. For instance, on the Credit dataset with the evidence counterfactual expert, CFSC achieved a combined F1 score of 0.785, compared to the next best baseline (ATT-FL) at 0.735.
- Improved feature selection accuracy: The paper shows that CFSC is better at emulating the ground-truth instance-level feature selections, particularly in identifying "top groups" of instances with similar feature usage patterns.
- Density analysis: CFSC's predicted feature usage density was often closer to the true density compared to baselines, indicating more realistic and parsimonious feature selection.
- ATT-FL's limitations: While ATT-FL was a strong baseline, it sometimes struggled with feature selection, particularly in cases where the ground truth involved fewer features.
Limitations and Future Work
- Simulated Experts: The authors acknowledge that while simulated experts are useful, their findings might not perfectly translate to real-world scenarios with actual human annotators. They mitigated this by using two different expert simulation strategies and diverse datasets.
- Data Availability: The paper highlights the scarcity of tabular datasets with instance-level feature labels, which is a significant barrier to research in this area.
- Future Work: The authors suggest that future work could explore more sophisticated methods for simulating experts or directly learning from human feedback. They also note that the optimal balance between classification and feature selection performance (controlled by the $\gamma_a$ parameter) depends on the specific application and stakeholder needs.
Summary updated