On Whom Should I Perform this Lab Test Next? An Active Feature Elicitation Approach

Sriraam Natarajan, Srijita Das, Nandini Ramanan, Gautam Kunapuli, Predrag RadivojacIJCAI 2018 · DOI

TL;DR

This paper introduces "Active Feature Elicitation" (AFE), a novel approach to intelligently select which examples to gather more data from, particularly when some features are expensive or difficult to obtain. AFE aims to improve machine learning model performance by strategically acquiring the most informative missing features, demonstrating significant gains in recall and overall performance across various medical datasets.

What Problem Does This Paper Address?

In many real-world scenarios, especially in medicine, collecting complete data for every example is not feasible. Some features are more expensive, time-consuming, or invasive to acquire than others. This creates a challenge for building robust machine learning models: how do you decide which examples are most worth the effort to gather additional, costly information from? Traditional active learning focuses on acquiring labels for entire examples, but this paper tackles the problem of acquiring specific features for already labeled or partially labeled examples to maximize model improvement. This is crucial for applications like clinical studies where recruiting participants for expensive tests (like fMRI scans or detailed lab work) needs to be highly targeted.

Methods

The core of the paper is the Active Feature Elicitation (AFE) approach. At each iteration, AFE works as follows:

  1. Identify Dissimilar Examples: AFE identifies examples in the partially observed dataset that are most "different" from the examples with fully observed features. This difference is quantified using a distance metric, specifically the Kullback-Leibler (KL) divergence. Crucially, this divergence is computed using models trained on both the fully observed features and the partially observed features, ensuring that the "informativeness" considers the full picture.
  2. Query for Missing Features: The top n most dissimilar examples are selected. For these selected examples, the missing features (e.g., lab test results) are then actively queried and collected.
  3. Update and Retrain: The newly acquired features and their corresponding examples are added to the fully observed dataset. The machine learning model is then retrained or updated with this augmented dataset.
  4. Iterate: This process repeats until a convergence criterion is met (e.g., the improvement in model performance plateaus).

AFE is designed to be classifier-independent and distance-metric independent, making it a general framework. The authors use gradient boosting as their classifier and KL-divergence for distance.

Datasets

The paper evaluates AFE on five datasets:

  • ADNI (Alzheimer's Disease NeuroInitiative): Predicting Alzheimer's progression using demographics (observed) and cognitive scores/fMRI (unobserved).
  • PPMI (Parkinson's Progression Markers Initiative): Predicting Parkinson's disease using demographics (observed) and motor/non-motor assessments (unobserved).
  • Rare Disease: Predicting rare diseases from survey data, using demographics (observed) and other survey answers (unobserved).
  • PPD (Post-Partum Depression): Predicting PPD using demographics (observed) and sensitive questionnaire data (unobserved).
  • PIMA Indians Diabetes Dataset: A standard UCI dataset used to demonstrate generality.

Key Insights and Ideas

  • Strategic Feature Acquisition is Key: Simply collecting more data randomly is inefficient. AFE's intelligent selection of examples for feature acquisition leads to significant performance gains.
  • Dissimilarity as a Proxy for Informativeness: Examples that are most different from the existing fully observed data are likely to provide the most novel information when their missing features are acquired.
  • Leveraging Existing Data: AFE effectively uses a smaller set of fully observed examples to guide the acquisition of features for a larger set of partially observed examples.
  • Robustness to Class Imbalance: The approach shows strong performance, particularly in recall, on imbalanced datasets, which is a common challenge in medical applications.
  • Generalizability: AFE's framework is flexible, allowing for different classifiers and distance metrics, and has been shown to work across diverse medical and general datasets.

Key Results

The paper demonstrates that AFE consistently outperforms baseline methods, including random selection (RND), uncertainty sampling on partially observed features (USObs), and uncertainty sampling with imputation (USAll).

  • Recall Improvement: AFE significantly improves recall, especially in the early stages of active learning, which is critical for identifying positive cases in imbalanced medical datasets. For example, in ADNI and PPD, AFE achieved recall over 0.8 with just a few iterations.
  • Overall Performance: AFE also shows strong performance in F1-score and geometric mean, indicating a good balance between precision and recall without sacrificing overall performance.
  • Reduced Variance: AFE exhibits lower variance in performance across different runs and datasets compared to random selection, suggesting more reliable and consistent results.

Limitations and Future Work

  • Current Feature Acquisition: AFE currently queries all missing features for selected examples. Future work could involve identifying the most informative subset of features for each example, rather than acquiring all of them.
  • Personalized Medicine: Extending the framework to identify groups of examples (sub-populations) that would provide the most information collectively, or to handle fully relational/graph data, are suggested future directions.
  • Imputation Techniques: While AFE outperforms imputation-based baselines, further research into more advanced imputation methods could potentially improve those baselines.
  • Convergence Criteria: While the paper uses log-likelihood difference, other convergence criteria could be explored.

Summary updated

BibTeX
@inproceedings{natarajan2018whom,
  author = {Sriraam Natarajan and Srijita Das and Nandini Ramanan and Gautam Kunapuli and Predrag Radivojac},
  title = {{On Whom Should I Perform this Lab Test Next? An Active Feature Elicitation Approach}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/486},
}
Figure