Bayesian Double Feature Allocation for Phenotyping With Electronic Health Records

Yang Ni, Peter Müller, Yuan JiJournal of the American Statistical Association 2018

Abstract

We propose a categorical matrix factorization method to infer latent diseases from electronic health records (EHR) data in an unsupervised manner. A latent disease is defined as an unknown biological aberration that causes a set of common symptoms for a group of patients. The proposed approach is based on a novel double feature allocation model which simultaneously allocates features to the rows and the columns of a categorical matrix. Using a Bayesian approach, available prior information on known diseases greatly improves identifiability and interpretability of latent diseases. This includes known diagnoses for patients and known association of diseases with symptoms. We validate the proposed approach by simulation studies including mis-specified models and comparison with sparse latent factor models. In the application to Chinese EHR data, we find interesting results, some of which agree with related clinical and medical knowledge.

Introduction

Phenotyping—the process of identifying a patient’s medical conditions from their data—is a cornerstone of modern healthcare. However, Electronic Health Records (EHR) are notoriously messy. Most existing methods for "disease mining" suffer from two major flaws. First, traditional clustering often assumes a patient has only one disease or that a symptom belongs to only one condition. In reality, patients often have comorbidities, and different diseases can share the same symptoms. Second, many matrix factorization techniques are built for continuous, normally distributed data, whereas EHR data, particularly lab results, are often categorical (e.g., "low," "normal," or "high" relative to a reference range).

This paper introduces a Bayesian Double Feature Allocation (DFA) model designed to discover "latent diseases" from categorical EHR data. By simultaneously clustering patients and symptoms into overlapping groups, the model provides an unsupervised framework that identifies complex many-to-many medical relationships while seamlessly incorporating existing clinical knowledge.

Methodology

The core of the research is the Double Feature Allocation (DFA) model. It treats phenotyping as a categorical matrix factorization task, where an observed patient-symptom matrix is decomposed into two latent binary/ternary matrices:

  • Patient-Disease (PD) Matrix: Links patients to latent diseases.
  • Symptom-Disease (SD) Matrix: Links symptoms to those same latent diseases.

The model utilizes the Indian Buffet Process (IBP) as a prior for patient-disease relationships, allowing it to learn the optimal number of diseases from the data nonparametrically. Unlike "black-box" unsupervised models, DFA allows researchers to "anchor" the model by fixing certain columns with known information, such as existing diagnoses or known medical associations (e.g., "Diabetes causes high glucose"). Inference is performed using Markov Chain Monte Carlo (MCMC) for posterior simulation, providing a full probabilistic understanding of the results.

Experimental Results

The researchers evaluated the DFA model using both simulated data and a real-world dataset of 1,000 adults from physical examinations in northeast China, including 39 blood test items. In controlled simulation tests, the model correctly identified the number of latent diseases in 96% of trials and recovered the underlying relationship matrices with error rates as low as 1-3%.

When applied to the Chinese EHR data, the model identified 14 diseases in total. This included the 4 "known" diseases (Diabetes, Kidney Disease, Liver Disease, Hypertension) and 10 newly discovered latent phenotypes. Several of these latent phenotypes mapped directly to recognized medical conditions, such as lipid disorder, anemia, thrombocytopenia, and viral/bacterial infections.

Visualization of the inferred Symptom-Disease (a), Patient-Disease (b), and data matrices (c, d) for the Chinese EHR dataset.

The DFA outperformed Sparse Latent Factor Models (SLFM), which struggled to interpret the non-normal distribution of the data and produced "noisy" links, such as incorrectly associating uric acid with almost every cluster.

Discussion and Future Work

A key insight of this work is that discretizing raw continuous lab values into ternary categories (Low/Normal/High) naturally incorporates medical context and makes the model more robust to outliers. Furthermore, the "semi-unsupervised" approach—incorporating a small amount of prior knowledge—significantly improved the interpretability of the discovered latent diseases.

However, limitations remain. MCMC is computationally expensive, posing challenges for scaling to millions of records. Additionally, the model does not explicitly account for treatment confounders (e.g., medication normalizing blood pressure) or "informative missingness," where the absence of a test result itself carries clinical meaning. Future iterations could benefit from modeling these complex clinical dynamics.

Summary updated

Abstract

Electronic health records (EHR) provide opportunities for deeper understanding of human phenotypes - in our case, latent disease - based on statistical modeling. We propose a categorical matrix factorization method to infer latent diseases from EHR data. A latent disease is defined as an unknown biological aberration that causes a set of common symptoms for a group of patients. The proposed approach is based on a novel double feature allocation model which simultaneously allocates features to the rows and the columns of a categorical matrix. Using a Bayesian approach, available prior information on known diseases (e.g., hypertension and diabetes) greatly improves identifiability and interpretability of the latent diseases. We assess the proposed approach by simulation studies including mis-specified models and comparison with sparse latent factor models. In the application to a Chinese EHR data set, we identify 10 latent diseases, each of which is shared by groups of subjects with specific health traits related to lipid disorder, thrombocytopenia, polycythemia, anemia, bacterial and viral infections, allergy, and malnutrition. The identification of the latent diseases can help healthcare officials better monitor the subjects' ongoing health conditions and look into potential risk factors and approaches for disease prevention. We cross-check the reported latent diseases with medical literature and find agreement between our discovery and reported findings elsewhere. We provide an R package "dfa" implementing our method and an R shiny web application reporting the findings.
BibTeX
@article{ni2018bayesian,
  author = {Yang Ni and Peter Müller and Yuan Ji},
  title = {{Bayesian Double Feature Allocation for Phenotyping With Electronic Health Records}},
  journal = {Journal of the American Statistical Association},
  year = {2018},
}
Figure