An Investigation Into Feature Selection for Oncological Survival Prediction

Dmitry Strunkin, Brian Mac Namee, John D. Kelleher

Introduction

Clinical Decision Support (CDS) systems in oncology face the high-stakes challenge of accurately predicting patient survival. While traditional systems rely on structured metrics like age or tumor stage, medical records are often rich with unstructured "secondary" data—free-text discharge reports containing narrative details. This study investigates whether augmenting expert-designed structured features with information mined from unstructured text improves predictive power or if the inherent "noise" in human-written notes hinders model generalization.

Methods

The researchers evaluated five combinations of feature sets to determine the most effective approach for survival prediction:

  1. Designed only: 25 structured data points (e.g., age, operation type).
  2. Automatically extracted text only: A "bag-of-words" approach based on word frequency in text fields.
  3. Manually extracted text only: Specific medical features (e.g., relapse, metastasis) manually identified by students following oncological guidelines.
  4. Designed + Automatically extracted text.
  5. Designed + Manually extracted text.

Three machine learning algorithms—Naive Bayes, Voting Feature Intervals (VFI), and J48 Decision Trees—were tested using 10-fold cross-validation. Performance was measured primarily by "average class accuracy" to mitigate the impact of significant class imbalance in the dataset.

Dataset and Class Imbalance

The study utilized a real-world dataset of 843 anonymized discharge reports for kidney cancer patients treated between 2000 and 2010 at the Republican Clinical Oncologic Dispensary in Kazan, Russia. Each report contained 25 structured fields and three free-text fields (medical history, surgical, and diagnostic descriptions).

Table 1: Dataset characteristics for 1, 3, and 5-year survival observation periods, highlighting the class imbalance between survivors and non-survivors.

As shown in the dataset summary above, there is a notable class imbalance, with survivors significantly outnumbering those who died. This imbalance makes predicting the "Died" class particularly challenging for standard algorithms, as they may default to predicting survival for all patients to maintain high overall accuracy.

Results and Discussion

The experimental results consistently highlighted the superiority of expert-designed features over automated or manual text extraction.

The Primacy of Expert Knowledge

In nearly every test scenario, features designed by expert oncologists provided the strongest baseline. Interestingly, adding unstructured data—even when manually curated—often had a negative impact on accuracy. This suggests that the high variability in clinical notes (abbreviations, varying detail levels, and typography) introduces more noise than signal for current machine learning models.

Table 2: Comparative performance of different feature sets and algorithms for 1-year survival prediction.

Algorithm Performance

As detailed in the performance table, the Voting Feature Intervals (VFI) classifier proved to be the most robust, consistently outperforming Naive Bayes and Decision Trees. For 1-year survival, VFI using only designed features achieved an average class accuracy of 0.764. In contrast, models relying solely on automatically extracted "bag-of-words" features performed poorly, with accuracies near 0.515, barely exceeding random chance.

The Horizon Effect

Predictive accuracy decreased as the survival window extended from 1 year to 5 years. This "horizon effect" reflects the increasing complexity and number of external variables that influence long-term oncological outcomes, making them inherently harder to model than short-term survival.

Conclusion

This investigation underscores that in clinical survival prediction, "more data" is not synonymous with "better data." Expert-designed structured features remain the gold standard for high-stakes clinical predictions. Future research should explore interactive data understanding tools that keep domain experts "in-the-loop," potentially refining how features are extracted from unstructured sources to find a balance between raw automation and rigid structured data.

Summary updated

Abstract

In machine learning based clinical decision support (CDS) systems the features used to train prediction models are of paramount importance. Strong features will lead to accurate models, whereas as weak features will have the opposite effect. Feature sets can either be designed by domain experts, or automatically extracted for unstructured data that happens to be available from some process other than a CDS system. This paper compares the usefulness of structured expert-designed features to features extracted from unstructured data sources in an oncological survival prediction application scenario.
BibTeX
@inproceedings{strunkin2012investigation,
  author = {Dmitry Strunkin and Brian Mac Namee and John D. Kelleher},
  title = {{An Investigation Into Feature Selection for Oncological Survival Prediction}},
  booktitle = {ITNG},
  year = {2012},
  doi = {10.1109/itng.2012.148},
}
Figure