Covered Information Disentanglement: Model Transparency via Unbiased Permutation Importance

João P. B. Pereira, Erik S. G. Stroes, Aeilko H. Zwinderman, Evgeni LevinAAAI 2022 · DOI

Introduction

In high-stakes domains like medicine, model interpretability is often as critical as predictive accuracy. A standard tool for global feature importance is permutation importance (PI), which measures the drop in model performance when a feature's values are shuffled. However, PI is fundamentally flawed in the presence of multicollinearity. When features are highly correlated, one feature can "cover" for another during permutation, leading to underestimated importance scores. This "covering" effect obscures true underlying mechanisms, hindering scientific discovery.

Methodology: The CID Framework

To address this, the authors introduce Covered Information Disentanglement (CID). This framework quantifies and corrects for information overlap using several advanced techniques:

  1. Information Theory & PID: CID uses entropy and Partial Information Decomposition (PID) to separate a feature's information into unique, redundant (shared), and synergistic components.
  2. Markov Random Fields (MRFs): To overcome the computational complexity of multivariate information, the authors model data as an undirected graph. They prove that covered information can be efficiently computed using the graph's cliques.
  3. The CID Algorithm: The process involves inferring the feature network structure, computing "covered" and "uncovered" entropy, and learning a mapping function ($\phi$) to relate these values back to observed performance drops.

Conceptual Framework: The "Covering" Intuition

The core idea of CID can be visualized through information-theoretic overlaps. While standard PI only captures the "unique" information a feature provides, CID accounts for the entire information content, including parts shared with other features.

Venn diagram illustrating the concept of covered information and unique information.

By treating feature importance like a Venn diagram, CID identifies how much of a feature's contribution is hidden by its neighbors in the feature space.

Experimental Setup

The framework was evaluated on two benchmarks:

  • Synthetic Dataset: A toy dataset with 2,000 samples and seven features ($X_1$ to $X_7$) with a known ground-truth importance ranking ($I_1 \ge I_2 \ge \dots \ge I_7$).
  • Cardiovascular Proteomics: A real-world dataset of 822 individuals using 50 proteins to predict acute myocardial infarction risk. This dataset is characterized by high correlation among proteins in shared biological pathways.

Results and Validation

Recovering Ground Truth

On the synthetic dataset, standard PI failed significantly, assigning near-zero importance to all features due to high covariance. In contrast, CID successfully recovered the exact ground-truth ranking.

Comparison of feature importance rankings on a synthetic dataset.

Medical Validation

In the cardiovascular study, CID elevated the importance of proteins like TRAIL-R2 and GDF-15. Medical experts confirmed these as high-quality predictors of long-term mortality, suggesting CID uncovers biologically relevant signals that standard methods miss. Quantitatively, CID outperformed PI, SAGE, and Tree-based methods in correlating feature importance sums with actual model performance on feature subsets.

Limitations and Future Work

While powerful, CID currently assumes Gaussian Markov Random Fields, which may not capture complex non-linear dependencies. The accuracy also relies on the consistency of the mapping function $\phi$. Future research will explore non-parametric MRFs and graph-based neural networks to enhance flexibility and modeling of the relationship between entropy and performance loss.

Summary updated

BibTeX
@inproceedings{pereira2022covered,
  author = {João P. B. Pereira and Erik S. G. Stroes and Aeilko H. Zwinderman and Evgeni Levin},
  title = {{Covered Information Disentanglement: Model Transparency via Unbiased Permutation Importance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i7.20769},
}
Figure