Introduction
Clinical Decision Support (CDS) systems in oncology face the high-stakes challenge of accurately predicting patient survival. While traditional systems rely on structured metrics like age or tumor stage, medical records are often rich with unstructured "secondary" data—free-text discharge reports containing narrative details. This study investigates whether augmenting expert-designed structured features with information mined from unstructured text improves predictive power or if the inherent "noise" in human-written notes hinders model generalization.
Methods
The researchers evaluated five combinations of feature sets to determine the most effective approach for survival prediction:
- Designed only: 25 structured data points (e.g., age, operation type).
- Automatically extracted text only: A "bag-of-words" approach based on word frequency in text fields.
- Manually extracted text only: Specific medical features (e.g., relapse, metastasis) manually identified by students following oncological guidelines.
- Designed + Automatically extracted text.
- Designed + Manually extracted text.
Three machine learning algorithms—Naive Bayes, Voting Feature Intervals (VFI), and J48 Decision Trees—were tested using 10-fold cross-validation. Performance was measured primarily by "average class accuracy" to mitigate the impact of significant class imbalance in the dataset.
Dataset and Class Imbalance
The study utilized a real-world dataset of 843 anonymized discharge reports for kidney cancer patients treated between 2000 and 2010 at the Republican Clinical Oncologic Dispensary in Kazan, Russia. Each report contained 25 structured fields and three free-text fields (medical history, surgical, and diagnostic descriptions).
As shown in the dataset summary above, there is a notable class imbalance, with survivors significantly outnumbering those who died. This imbalance makes predicting the "Died" class particularly challenging for standard algorithms, as they may default to predicting survival for all patients to maintain high overall accuracy.
Results and Discussion
The experimental results consistently highlighted the superiority of expert-designed features over automated or manual text extraction.
The Primacy of Expert Knowledge
In nearly every test scenario, features designed by expert oncologists provided the strongest baseline. Interestingly, adding unstructured data—even when manually curated—often had a negative impact on accuracy. This suggests that the high variability in clinical notes (abbreviations, varying detail levels, and typography) introduces more noise than signal for current machine learning models.
Algorithm Performance
As detailed in the performance table, the Voting Feature Intervals (VFI) classifier proved to be the most robust, consistently outperforming Naive Bayes and Decision Trees. For 1-year survival, VFI using only designed features achieved an average class accuracy of 0.764. In contrast, models relying solely on automatically extracted "bag-of-words" features performed poorly, with accuracies near 0.515, barely exceeding random chance.
The Horizon Effect
Predictive accuracy decreased as the survival window extended from 1 year to 5 years. This "horizon effect" reflects the increasing complexity and number of external variables that influence long-term oncological outcomes, making them inherently harder to model than short-term survival.
Conclusion
This investigation underscores that in clinical survival prediction, "more data" is not synonymous with "better data." Expert-designed structured features remain the gold standard for high-stakes clinical predictions. Future research should explore interactive data understanding tools that keep domain experts "in-the-loop," potentially refining how features are extracted from unstructured sources to find a balance between raw automation and rigid structured data.
Summary updated