Feature Disentanglement through Mutual Information Minimization

This direction is merged into another direction: mutual information minimization (unsupervised factor disentanglement). Its last version stays readable and citable here.

Version 1 of 124 papers28 conceptsfrozen 2026-10-04

When a neural network learns a representation of data, it often creates a "tangled" mess of features. For example, a face recognition model might accidentally bake a person’s age or gender into the same numerical vector used to identify their name. In multi-view learning—where a model looks at the same object from different camera angles—the representation of the "object" often gets polluted by the "angle."

To solve this, a growing body of research focuses on disentanglement: the art of forcing a model to partition its knowledge into distinct, non-overlapping subspaces. The primary tool for achieving this is the minimization of Mutual Information (MI). By treating different feature sets as random variables and actively penalizing the information they share, researchers are building models that are more robust, interpretable, and fair.

The Shared and the Specific

The most common application of this idea is in multi-view learning. Imagine you have a video of a car and a thermal image of the same car. There is "shared" information (it’s a car, it’s moving fast) and "view-specific" information (the color of the paint vs. the heat of the engine).

Standard models often struggle to separate these, leading to what Long Shi and colleagues call the "redundancy assumption"—the mistaken belief that only shared information matters. In Towards Comprehensive Information-theoretic Multi-view Learning, the authors argue that we must extract both. To ensure these representations don't just copy each other, they minimize the MI between the shared and specific feature sets.

This "untangling" acts as a filter. As Guanzhou Ke demonstrates in Rethinking Multi-View Representation Learning via Distilled Disentangling, penalizing redundancy ensures that view-specific encoders only retain information that the shared consistency representation hasn't already captured.

Preventing Information Leakage

Disentanglement isn't just about organization; it’s about "purification." In many systems, we want to ensure that one part of the model doesn't "know" what another part is doing.

  • Fairness: In DebFace: De-biasing Face Recognition, Sixue Gong and Anil K. Jain 0001 use a factorization loss to minimize MI between identity features and demographic traits. This prevents the model from using race or gender as a "shortcut" for recognition, reducing systemic bias.
  • Interpretability: Concept Bottleneck Models try to explain their decisions using human-readable concepts (e.g., "this bird has a red wing"). However, if the model has a "residual" side-channel, it might sneak uninterpretable data through it. Renos Zabounidis and Ini Oguntola show in Benchmarking and Enhancing Disentanglement in Concept-Residual Models that MI minimization prevents this "leakage," forcing the residual to only capture what the concepts truly cannot.
  • Federated Learning: When training models across many private devices, local "noise" from one user can pollute the global model. Yongqiang Huang introduced FedRIR, which uses MI minimization to decouple global knowledge from local patterns, ensuring the shared model stays generalizable.

How MI Minimization Works Intuitively

If you want two variables, $A$ and $B$, to be independent, you want to reduce the amount of "surprise" that knowing $A$ provides about $B$.

In practice, calculating exact Mutual Information is computationally difficult for high-dimensional neural network features. Instead, researchers use "proxies" or bounds. For instance, David Restrepo in the DF-DM framework uses a variational upper bound. This acts like a mathematical ceiling; by pushing the ceiling down, you force the MI to decrease.

Another popular method is Mutual Information Neural Estimation (MINE). As seen in the work of Yanxi Li on Vision MLPs, a small "critic" network is trained to try and tell the difference between paired features (from the same image) and shuffled features (from different images). If the critic can't tell them apart, the MI is low, and the features are successfully disentangled.

Specialization through Competition

Finally, MI minimization is used to force "experts" to specialize. In MoMoK, Yichi Zhang and Lingbing Guo use a disentanglement loss to ensure that different sub-networks (experts) focus on distinct features of a multi-modal knowledge graph.

By penalizing the shared information between these experts, the model prevents "information collapse"—a failure mode where every part of the network learns the same easy pattern. Instead, each expert is forced to find a unique, non-redundant niche, leading to a more diverse and robust "committee" of features.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feature_disentanglement_through_mutual_i_b2dcb09e,
  title = {Feature Disentanglement through Mutual Information Minimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-disentanglement-through-mutual-information-minimization/rdr_b2dcb09e38d1465c85fcb9b9ca59d576},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_feature_disentanglement_through_mutual_i_b2dcb09e_v1,
  title = {Feature Disentanglement through Mutual Information Minimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-disentanglement-through-mutual-information-minimization/rdr_b2dcb09e38d1465c85fcb9b9ca59d576/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}