Abstract
In high-stakes domains like healthcare, there is a persistent tension between the interpretability of Probabilistic Graphical Models (PGMs) and the predictive power of deep neural networks. This paper introduces a constrained variational inference framework that distills the discriminative capabilities of "black-box" teacher models into interpretable PGM students. By incorporating a similarity-preserving constraint into the Evidence Lower Bound (ELBO), the proposed method produces models that achieve high predictive accuracy on clinical tasks without sacrificing generative quality.
Introduction
Researchers in healthcare often face a trade-off: use interpretable models like topic models or Hidden Markov Models (HMMs) that offer clear latent structures (e.g., disease subtypes), or use deep learning models that excel at prediction but remain opaque. Standard PGMs often underperform on downstream prediction tasks when their latent variables are used as features. Conversely, deep neural networks provide little insight into the underlying data structure. This work bridges the gap by "teaching" interpretable graphical models to represent data similarly to powerful discriminative models.
Methodology: Constrained Variational Inference
The authors propose a Teacher-Student framework where a high-performing discriminative model (the teacher) guides a PGM (the student).
The core innovation is the Similarity-Preserving Constraint. Instead of matching raw output scores—which is challenging due to architectural differences—the framework matches the pairwise similarity of feature representations. If the teacher model identifies two patients as similar, the constraint encourages the student PGM's latent variables to reflect that same relationship.
This constraint is added as a penalty term to the standard Evidence Lower Bound (ELBO). The optimization objective balances maximizing the likelihood of observed data (generative quality) with minimizing the discrepancy between the teacher's and student's similarity matrices. The framework utilizes Automatic Differentiation Variational Inference (ADVI) and Autoencoding Variational Bayes (AEVB) to maintain flexibility across various PGM architectures.
Experimental Setup
The framework was evaluated on two complex medical tasks:
- COPDGene: A large-scale dataset of lung CT images from 7,292 subjects. The goal was to identify disease subtypes and predict clinical measurements like lung function (FEV1pp).
- MIMIC-III: A public critical care database. The authors focused on a cohort of 11,648 sepsis patients, using 29 time-varying physiological variables to monitor disease progression and predict in-hospital mortality.
Results and Discussion
The results demonstrate that matching the geometry of the feature space is a robust distillation strategy.
As shown in the similarity matrix comparisons, the distilled model (KD-LDA) successfully captures the structural relationships identified by the teacher model far more effectively than unsupervised or standard supervised baselines.
Key Performance Metrics
- Disease Subtyping (COPD): KD-LDA achieved an $R^2$ of 0.49 for lung function prediction, more than tripling the performance of the unsupervised baseline (0.16) and significantly beating a standard supervised PGM (0.29).
- Sepsis Monitoring: For mortality prediction, the distilled AR-HMM (KD-ARHMM) achieved an AUROC of 0.65, approaching the teacher LSTM's performance (0.71) and far outperforming the unsupervised baseline (0.56).
- Clinical Grounding: In the sepsis experiment, the distilled states were more "clinically meaningful," correlating much more strongly with the SOFA score (a standard clinical measure of organ failure) than the unsupervised version.
Importantly, the generative fit (ELBO) remained stable or improved across experiments, confirming that the distillation constraint doesn't "break" the underlying graphical model.
Conclusion
This work provides a flexible method for enhancing the predictive utility of interpretable models through knowledge distillation. While the framework requires manual tuning of the distillation constraint weight and remains subject to the typical challenges of variational inference, it represents a significant step toward deploying models that are both powerful and transparent in clinical settings. Future research may focus on automating hyperparameter selection and closing the "amortization gap" in black-box inference.
Summary updated