Ensembles of randomized trees using diverse distributed representations of clinical events

Aron Henriksson, Jing Zhao, Hercules Dalianis, Henrik Boström

Electronic Health Records (EHRs) represent a massive repository of clinical knowledge, yet extracting predictive insights from them remains a significant challenge. The data is characterized by high dimensionality, extreme sparsity, and heterogeneity, combining structured codes (like ICD-10) with unstructured clinical narratives. Traditional machine learning approaches often struggle with this "curse of dimensionality," where the vast majority of potential features for any given patient are absent.

In a study published in BMC Medical Informatics and Decision Making, Henriksson et al. propose a novel approach to this problem. By leveraging distributed representations of clinical events and integrating them into a specialized ensemble of randomized trees, the researchers demonstrate a significant improvement in detecting Adverse Drug Events (ADEs).

The Challenge of EHR Embeddings

To address data sparsity, researchers often use distributional semantics—techniques like Word2Vec—to map clinical events into dense, low-dimensional vectors. These embeddings ensure that semantically similar events (e.g., two different medications for the same condition) are positioned close to one another in a vector space.

However, the "meaning" captured by these embeddings depends heavily on the context window used during training. A small window might capture functional similarities, while a larger window captures broader clinical associations. Instead of searching for a single "optimal" window size, Henriksson et al. asked: can we build a model that utilizes multiple diverse representations simultaneously?

Methodology: RDR-ALL

The researchers evaluated three primary strategies for combining ten different semantic spaces, generated by varying context window sizes from 2 to 20:

  1. Fused Diverse Representations (FDR): A baseline approach where all ten representations are concatenated into a single large feature vector for a standard Random Forest.
  2. Randomized Diverse Representations with Feature Subsampling (RDR-FS): For each tree in the ensemble, one embedding type is randomly selected, followed by the standard Random Forest practice of selecting a random subset of those features.
  3. Randomized Diverse Representations without Feature Subsampling (RDR-ALL): The proposed method. Each tree is assigned one randomly selected embedding type but is permitted to use the entire feature set of that representation.

The logic behind RDR-ALL is to allow individual trees to exploit the full set of interactions within a specific semantic view of the data, such as the relationship between a specific drug code and a lab measurement.

Algorithm 1: The pseudocode for the proposed ensemble method using diverse representations.

Experimental Results

The methods were tested on 27 real-world clinical datasets from the Stockholm EPR Corpus, covering approximately 700,000 patients. The task was binary classification to detect specific ADEs. The corpus included 3 million unique words, over 9,000 diagnosis codes, and 1,272 drug codes.

The results showed that RDR-ALL was the superior strategy, achieving a mean accuracy of 89.11%. This outperformed both the concatenation baseline (88.08%) and the subspacing method (87.55%). While the improvement in AUC was more modest (0.942 vs 0.939), the accuracy gains were statistically significant ($p = 0.0023$).

The Accuracy-Diversity Trade-off

A key finding of the study involves the relationship between tree accuracy and ensemble diversity. In typical ensemble learning, diversity (trees making different errors) is prioritized. Interestingly, RDR-ALL produced trees that were less diverse than those in RDR-FS. However, the individual trees in RDR-ALL were significantly more accurate (76.02% vs 70.99%). This suggests that in the context of high-dimensional EHR embeddings, the gain in individual tree strength from seeing the full feature set outweighs the loss in diversity.

Conclusion and Future Directions

The study demonstrates that "diversity in representation" is a potent tool for clinical predictive modeling. By allowing different trees to specialize in different semantic "views" of the patient record, RDR-ALL effectively overcomes the limitations of data sparsity. Future research may expand this framework by incorporating diverse embedding algorithms (such as GloVe or FastText) or varying the dimensionality of the vector spaces to further enhance model robustness.

Summary updated

BibTeX
@article{henriksson2016ensembles,
  author = {Aron Henriksson and Jing Zhao and Hercules Dalianis and Henrik Boström},
  title = {{Ensembles of randomized trees using diverse distributed representations of clinical events}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2016},
  doi = {10.1186/s12911-016-0309-0},
}
Figure