Abstract
Dynamic feature selection, where we sequentially query features to make accurate predictions with a minimal budget, is a promising paradigm to reduce feature acquisition costs and provide transparency into the prediction process. The problem is challenging, however, as it requires both making predictions with arbitrary feature sets and learning a policy to identify the most valuable selections. Here, we take an information-theoretic perspective and prioritize features based on their mutual information with the response variable. The main challenge is learning this selection policy, and we design a straightforward new modeling approach that estimates the mutual information in a discriminative rather than generative fashion. Building on our learning approach, we introduce several further improvements: allowing variable feature budgets across samples, enabling non-uniform costs between features, incorporating prior information, and exploring modern architectures to handle partial input information. We find that our method provides consistent gains over recent state-of-the-art methods across a variety of datasets.
Introduction
In critical domains such as medical diagnostics or high-resolution satellite imaging, data collection is often the most significant bottleneck. A clinician does not order every possible laboratory test simultaneously; instead, they sequentially select the next most informative test based on the current patient state. This process is known as Dynamic Feature Selection (DFS): the goal is to sequentially query features to achieve high predictive accuracy while minimizing a total acquisition budget.
Developing an effective DFS policy is difficult. Existing approaches typically utilize Reinforcement Learning (RL), which can be unstable and difficult to train, or information-theoretic greedy methods. The latter prioritize features with the highest Conditional Mutual Information (CMI) with the target variable. However, calculating CMI traditionally requires complex generative models that are computationally expensive and difficult to scale. This paper introduces DIME (Discriminative Mutual Information Estimation), a framework that estimates CMI through a simple discriminative objective.
Methodology: The DIME Framework
DIME employs two neural networks trained jointly to optimize the feature selection process:
- The Predictor ($f$): A model trained to make predictions (e.g., classification) using any arbitrary subset of available features.
- The Value Network ($v$): A model that estimates the CMI for every unobserved feature given the current subset.
The Discriminative Insight
The core mathematical contribution of DIME is the realization that if a predictor is trained with standard cross-entropy loss, the incremental reduction in loss when adding a new feature serves as an unbiased estimator of the CMI. Instead of modeling the entire data distribution, DIME trains the Value Network to solve a regression problem: predicting how much the classification loss will drop if a specific feature is added.
This approach offers several advantages:
- Non-uniform Costs: DIME can prioritize features by maximizing the ratio of information gain to cost ($CMI/c_i$).
- Flexible Budgets: The model can stop querying once the estimated information gain falls below a threshold ($\lambda$), allowing for sample-specific budgets.
- Architectural Flexibility: DIME is compatible with modern architectures like Vision Transformers (ViTs), which are naturally adept at handling masked or partial inputs via self-attention.
Experimental Results
The authors evaluated DIME across diverse benchmarks, including tabular medical datasets (Intubation, ROSMAP), image classification (MNIST), and patch-based image selection (Imagenette, MHIST).
Key Performance Insights
- State-of-the-Art Tradeoffs: As shown in the comparative results, DIME consistently achieves better accuracy-cost tradeoffs than RL baselines and previous discriminative methods like "Argmax Direct."
- High Efficiency: On the Imagenette dataset, DIME achieved approximately 97% accuracy using only 15 out of 196 patches—less than 8% of the total image data.
- Cost-Sensitivity: In medical tasks with non-uniform costs (e.g., comparing a cheap blood pressure reading to an expensive MRI), DIME successfully prioritized high-value, low-cost information.
- ViT Superiority: The research found that Vision Transformers significantly outperform CNNs for dynamic selection. Their self-attention mechanism is inherently better at integrating information from scattered, partial patches.
Conclusion
DIME demonstrates that you do not need to model the full probability distribution of features to determine their utility. By focusing on loss reduction, DIME bypasses the complexity of generative modeling while providing a robust, scalable framework for dynamic feature selection. While the greedy nature of the algorithm may face challenges with complex feature interactions (like the XOR problem), its performance across a wide array of real-world datasets establishes it as a powerful tool for cost-efficient machine learning.
Summary updated