Abstract
We propose a categorical matrix factorization method to infer latent diseases from electronic health records (EHR) data in an unsupervised manner. A latent disease is defined as an unknown biological aberration that causes a set of common symptoms for a group of patients. The proposed approach is based on a novel double feature allocation model which simultaneously allocates features to the rows and the columns of a categorical matrix. Using a Bayesian approach, available prior information on known diseases greatly improves identifiability and interpretability of latent diseases. This includes known diagnoses for patients and known association of diseases with symptoms. We validate the proposed approach by simulation studies including mis-specified models and comparison with sparse latent factor models. In the application to Chinese EHR data, we find interesting results, some of which agree with related clinical and medical knowledge.
Introduction
Phenotyping—the process of identifying a patient’s medical conditions from their data—is a cornerstone of modern healthcare. However, Electronic Health Records (EHR) are notoriously messy. Most existing methods for "disease mining" suffer from two major flaws. First, traditional clustering often assumes a patient has only one disease or that a symptom belongs to only one condition. In reality, patients often have comorbidities, and different diseases can share the same symptoms. Second, many matrix factorization techniques are built for continuous, normally distributed data, whereas EHR data, particularly lab results, are often categorical (e.g., "low," "normal," or "high" relative to a reference range).
This paper introduces a Bayesian Double Feature Allocation (DFA) model designed to discover "latent diseases" from categorical EHR data. By simultaneously clustering patients and symptoms into overlapping groups, the model provides an unsupervised framework that identifies complex many-to-many medical relationships while seamlessly incorporating existing clinical knowledge.
Methodology
The core of the research is the Double Feature Allocation (DFA) model. It treats phenotyping as a categorical matrix factorization task, where an observed patient-symptom matrix is decomposed into two latent binary/ternary matrices:
- Patient-Disease (PD) Matrix: Links patients to latent diseases.
- Symptom-Disease (SD) Matrix: Links symptoms to those same latent diseases.
The model utilizes the Indian Buffet Process (IBP) as a prior for patient-disease relationships, allowing it to learn the optimal number of diseases from the data nonparametrically. Unlike "black-box" unsupervised models, DFA allows researchers to "anchor" the model by fixing certain columns with known information, such as existing diagnoses or known medical associations (e.g., "Diabetes causes high glucose"). Inference is performed using Markov Chain Monte Carlo (MCMC) for posterior simulation, providing a full probabilistic understanding of the results.
Experimental Results
The researchers evaluated the DFA model using both simulated data and a real-world dataset of 1,000 adults from physical examinations in northeast China, including 39 blood test items. In controlled simulation tests, the model correctly identified the number of latent diseases in 96% of trials and recovered the underlying relationship matrices with error rates as low as 1-3%.
When applied to the Chinese EHR data, the model identified 14 diseases in total. This included the 4 "known" diseases (Diabetes, Kidney Disease, Liver Disease, Hypertension) and 10 newly discovered latent phenotypes. Several of these latent phenotypes mapped directly to recognized medical conditions, such as lipid disorder, anemia, thrombocytopenia, and viral/bacterial infections.
The DFA outperformed Sparse Latent Factor Models (SLFM), which struggled to interpret the non-normal distribution of the data and produced "noisy" links, such as incorrectly associating uric acid with almost every cluster.
Discussion and Future Work
A key insight of this work is that discretizing raw continuous lab values into ternary categories (Low/Normal/High) naturally incorporates medical context and makes the model more robust to outliers. Furthermore, the "semi-unsupervised" approach—incorporating a small amount of prior knowledge—significantly improved the interpretability of the discovered latent diseases.
However, limitations remain. MCMC is computationally expensive, posing challenges for scaling to millions of records. Additionally, the model does not explicitly account for treatment confounders (e.g., medication normalizing blood pressure) or "informative missingness," where the absence of a test result itself carries clinical meaning. Future iterations could benefit from modeling these complex clinical dynamics.
Summary updated