Introduction
Understanding the longitudinal sequence of cancer treatments is critical for evaluating therapeutic effectiveness. However, this information is often buried within Electronic Medical Records (EMRs). While cancer registries exist, they are frequently limited to initial treatments and rely on labor-intensive manual abstraction. This research presents a machine learning pipeline designed to automate treatment classification by integrating structured patient data with unstructured clinical notes, achieving classification accuracies exceeding 90%.
Methodology
The proposed system utilizes a two-stage feature fusion approach to process heterogeneous medical data:
- Unsupervised Text Embedding: Unstructured clinical notes are transformed into numerical vectors using a doc2vec model. This model was trained on a corpus of over 480,000 notes to capture the semantic nuances of clinical oncology. To reduce noise, the notes were pre-filtered for sentences containing specific treatment terms from the NCI metathesaurus.
- Feature Fusion: Structured features, including demographic and clinical data, are concatenated with the doc2vec embeddings to create a unified feature vector. The specific features used are detailed in the table below.
- Classification: A logistic regression classifier is applied to the combined vector. The researchers conducted extensive hyperparameter tuning, varying vector dimensionality, learning rates ($\alpha$), and training epochs to optimize performance.
Data Description
The study utilized data from two primary sources: the Stanford Cancer Institute Research Database (SCIRDB) for clinical notes and structured data, and the California Cancer Registry (CCR) for ground-truth treatment labels. The cohort included 2,389 patients diagnosed with prostate, esophageal, or oropharynx cancers after 2008.
Results and Discussion
The experimental results demonstrate the significant value of incorporating narrative clinical notes. A baseline model using only structured data achieved an accuracy of 87.24%. By integrating doc2vec embeddings, the peak accuracy rose to 90.59%.
Key insights from the tuning process revealed that the quality of the learning process (epochs and learning rate) was more influential than the absolute size of the embedding vector. Optimal results were consistently observed with vector sizes between 300 and 500. Furthermore, the "denoising" step—filtering notes for relevant medical terms—was instrumental in focusing the model on treatment-specific signals rather than general clinical discourse.
Conclusion and Future Work
This study confirms that NLP-processed clinical notes contain vital context missing from structured EMR fields. Future work will explore non-linear classification methods such as Random Forests and Convolutional Neural Networks. Additionally, the researchers plan to expand the feature set with ICD9 and CPT codes and investigate the potential for transfer learning across different cancer types to eventually track full longitudinal treatment sequences.
Summary updated