Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization

Joshua C. Chang, Patrick Fletcher, Jungmin Han, Ted L. Chang, Shashaank Vattikuti, Bart Desmet, Ayah Zirikly, Carson C. ChowICLR 2021

Overview

Researchers have developed a new "sparsely-encoded" version of Hierarchical Poisson Factorization (HPF) that explicitly links latent data representations back to specific input features. By enforcing sparsity in the encoder rather than just the decoder, the model achieves intrinsic interpretability and automatic feature selection. This ensures that learned factors are actually grounded in identifiable subsets of the original data, rather than an uninterpretable blend of all inputs.

The Interpretability Trap

In fields like medical informatics, understanding why a model makes a prediction is often as important as the prediction itself. Dimensionality reduction techniques like Non-negative Matrix Factorization (NMF) and Hierarchical Poisson Factorization (HPF) are popular because they are "generative"—they explain how latent factors (like "respiratory health") combine to produce observed data (like hospital billing codes).

However, there is a subtle but critical "interpretability trap." While these models are often sparse in their decoding (how factors predict data), they are typically dense in their encoding (how data forms factors). In standard HPF, every single input feature technically contributes to every latent factor. This makes it nearly impossible to say exactly which original features define a specific factor. Practitioners often mistakenly read the decoder matrix as if it were an encoder, leading to "post-hoc" explanations that can be mathematically incorrect and misleading.

Comparison of standard HPF and sparsely-encoded HPF pipelines, highlighting the interpretability of latent factors.

Proposed Method: Sparsely-Encoded HPF

The authors propose an augmented HPF model that incorporates an explicit encoder function based on a Generalized Additive Model (GAM).

Key technical innovations include:

  • Self-Consistent Constraints: The encoder transformation is learned simultaneously with the matrix factorization, ensuring the compressed representation is directly tied to the input features.
  • Horseshoe+ Prior: The model uses a non-negative version of this hierarchical Bayesian prior to enforce extreme sparsity in the encoding matrix. This allows the model to perform automatic feature selection, effectively "turning off" irrelevant features.
  • Background Rate Term: The model includes an item-specific "gain" term to capture independent noise. This prevents the factor model from trying to find structure in pure random noise.
  • Flexible Link Functions: Monotonic link functions are introduced to allow the model to handle non-linear relationships in the data without sacrificing the "more is more" interpretability of the factors.

Key Insights and Results

The model was evaluated on synthetic datasets (simulating pure noise, linear, and non-linear structures) as well as a massive real-world Medicare Limited Data Set (LDS) containing ~1.95 million inpatient visits across 136 clinical categories.

  • Decoder Sparsity $\neq$ Encoder Sparsity: Just because a model uses a few factors to reconstruct data doesn't mean those factors are built from a few features. This distinction is the core of the paper's argument for "intrinsic interpretability."
  • Feature Selection as a Guardrail: By forcing the encoder to be sparse, the model becomes highly robust to noise. In simulations of pure noise, the sparsely-encoded HPF correctly assigned features to the background rate and ignored them, while standard HPF hallucinated non-existent structures.
  • Clinical Relevance: In the Medicare study, the model identified a "lung ailment" factor. Interestingly, while "lung cancer" appeared only faintly in the standard decoder, it was strongly present in the new model's encoder. This demonstrates that rare but highly predictive features are better captured by sparse encoding.
  • Inverting Decision Rules: Because the encoder is an explicit function, a "high-risk" threshold in the latent factor space can be mathematically translated back into a concrete rule on the original features (e.g., "Factor 2 is high if the patient has more than X lung-related codes").
  • Maintained Performance: Using the Widely Applicable Information Criterion (WAIC), the authors showed that their more interpretable model maintained predictive accuracy comparable to standard, less interpretable versions.

Limitations and Future Work

While powerful, the approach has a few trade-offs:

  • Expressivity: As a GAM-based approach, it is inherently less flexible than deep learning. While it handles some non-linearity, it cannot capture complex, high-order interactions between features as well as a deep neural network.
  • Hyperparameter Sensitivity: The effectiveness of the Horseshoe+ prior depends on scaling. While the authors provide a robust default, manual tuning might be needed for specific levels of desired sparsity.
  • Computational Intensity: Bayesian inference via Automatic Differentiation Variational Inference (ADVI) is more computationally demanding than simple matrix factorization, though it still converges relatively quickly (approximately 100 epochs).

Summary updated

BibTeX
@inproceedings{chang2021sparse,
  author = {Joshua C. Chang and Patrick Fletcher and Jungmin Han and Ted L. Chang and Shashaank Vattikuti and Bart Desmet and Ayah Zirikly and Carson C. Chow},
  title = {{Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure