In multimodal machine learning, the traditional goal has been fusion: taking different data streams—like the pixels of an image and the text of a caption—and smashing them together into a single joint representation. However, as Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency argue in their foundational survey, Foundations and Recent Trends in Multimodal Machine Learning, this "black box" fusion often ignores a fundamental truth: different modalities are not just redundant versions of each other. They contain both shared information (the semantic core common to all inputs) and specific information (the unique nuances of a single sensor).
The emerging research direction of representation fission or disentanglement seeks to explicitly separate these two components. Instead of creating a "muddy brown" mixture of features, these models aim to keep the "primary colors" of each modality distinct while identifying where they overlap.
The Problem of Modality Suppression
Why not just let the neural network figure it out? When we fuse modalities naively, models often suffer from modality suppression. As Yi Wang and colleagues explain in Decoupling Common and Unique Representations for Multimodal Self-supervised Learning, if one modality is "easier" to learn from (like text in a sentiment task), the model might ignore the subtle cues in other modalities (like audio tone).
By forcing the model to extract unique features that cannot be found in other streams, researchers prevent the dominant modality from overshadowing the others. This ensures that the unique contribution of every data stream—such as the 3D structural cues in depth maps described by Aixuan Li in Mutual Information Regularization for Weakly-supervised RGB-D Salient Object Detection—is preserved and utilized.
Mechanisms of Disentanglement
To achieve this separation, researchers typically use two complementary strategies:
1. Architectural Partitioning
Models like MMSS, developed by Anran Wang and Jianfei Cai, project data into a joint feature space that is explicitly divided into "shared" and "specific" slots. Similarly, the ShaSpec framework by Congbo Ma uses dedicated encoders to ensure that modality-agnostic patterns (like the shape of a tumor) are stored separately from modality-unique signatures (like the specific contrast of an MRI vs. a CT scan).
2. Information-Theoretic Constraints
How do you "force" a feature to be unique? Researchers use Mutual Information (MI) as a mathematical wedge. In the SRCID framework, Hai Huang and Yan Xia minimize the mutual information between the shared and specific feature branches. If the MI is zero, the specific branch contains no information that can be predicted from the shared branch, effectively "cleaning" the shared representation of modality-specific noise.
Robustness and Missing Data
Disentanglement isn't just about accuracy; it's about robustness. In medical settings, a patient might have an MRI but lack a PET scan. Traditional fusion models often break when a modality is missing because they expect a complete, integrated signal.
By decoupling the representations, models become more modular. Min Gu Kwak and Jing Li demonstrate in A Cross-Modal Mutual Knowledge Distillation Framework that a model can learn to "hallucinate" or compensate for missing data more effectively if it already understands which parts of the information are shared across modalities and which are unique.
Beyond Fusion to Fission
Whether it is distinguishing "deepfake" manipulations by identifying mismatches between text and image Jiazhen Wang or matching person identities across visible and infrared cameras Yan Lu, the intuition remains the same: to truly understand multiple modalities, a model must first understand what makes each one unique. By shifting from "blind fusion" to "principled fission," the field is moving toward AI that is more interpretable, efficient, and resilient to the complexities of real-world data.