Heterogeneous Modality Encoding for Multimodal Deep Learning

This direction is split into other directions: heterogeneous modality encoding. Its last version stays readable and citable here.

Version 1 of 1167 papers170 conceptsfrozen 2026-10-04

In the early days of multimodal machine learning, the standard approach was "early fusion": flatten every input—whether it was a pixel, a sensor reading, or a word—into a single long vector and feed it into a generic neural network. But real-world data is rarely that uniform. A 3D point cloud of a kidney tumor has a fundamentally different mathematical structure than a patient’s clinical notes or a 1D sequence of heart rate readings.

Researchers have increasingly moved toward Heterogeneous Modality Encoding, a modular architecture where each data type is processed by a specialized "expert" encoder before being fused. This approach prioritizes preserving the structural integrity of each data source, ensuring that the geometric nuances of a 3D shape or the temporal dependencies of a weather simulation aren't lost in translation.

The Modular Encoder-Combiner-Decoder Pattern

The core intuition behind this direction is that a model's architecture should mirror the physics and geometry of its input. Piero Molino, Huaixiu Zheng, and Yi-Chia Wang formalized this with COTA, a system for customer support that uses specialized encoders for text, categorical metadata, and numerical values.

Instead of forcing these disparate signals into a single format, COTA uses a "Combiner" layer to integrate the outputs of these specialized encoders. This allows the model to "speak" the language of each data type natively. For example, a convolutional network might handle the spatial features of an image, while a recurrent unit handles the sequence of a chat history. By the time these signals reach the combiner, they have been compressed into high-level semantic representations that are finally compatible.

Tailoring Architectures to Data Geometry

The power of heterogeneous encoding lies in its ability to handle "domain gaps"—the massive differences in how different sensors perceive the same object. In Semantic Embedding for Sketch-Based 3D Shape Retrieval, Anran Qi and Yi-Zhe Song demonstrate that trying to force 2D sketches and 3D models into the same abstract space is less effective than using a dual-branch architecture. One branch specializes in the sparse, stroke-based nature of sketches, while the other handles the volumetric density of 3D shapes, only aligning them at the very end in a shared semantic label space.

This specialization is even more critical in scientific domains:

  • Molecular Modeling: In MOCO, Yanqiao Zhu and colleagues use four distinct encoders for a single molecule: GNNs for 2D topology, continuous-filter convolutions for 3D geometry, and Transformers for SMILES strings.
  • Medical Diagnostics: Bo Yang and Lijun Wu show that Electronic Health Records (EHR) are best processed by using MLPs for static demographics, Transformers for longitudinal vitals, and domain-specific LLMs for clinical notes.
  • Meteorology: Cheng Huang and Cong Bai developed MGTCF for cyclone forecasting, which uses 3D U-Nets for spatial atmospheric grids alongside 1D encoders for scalar attributes like wind speed.

Why Structure Beats Flattening

A recurring theme across these papers is that flattening data into a uniform format (like text) destroys information. While it is tempting to feed everything into a Large Language Model, Ziyi Gao and Yaoyao Xu argue in their work on InstructUE that preserving the structural integrity of tabular and sequential data through modality-specific adapters outperforms simple text tokenization.

When you flatten a table or a 3D point cloud into a string of text, you lose the "adjacency" information—the model no longer "knows" which points are close in 3D space or which cells are in the same column without significant extra learning. Specialized encoders, like the MixNet used for topographic maps in snowpack estimation, bake these inductive biases directly into the architecture.

Robustness and Hallucination

Beyond accuracy, heterogeneous encoding provides a safety net for missing data. Sayna Ebrahimi and Sercan O Arik introduced LANISTR, which uses similarity-based masking to remain robust even when entire modalities are missing. Because the encoders are modular, the system can still extract meaningful features from the remaining "experts."

In some cases, these specialized decoders can even "hallucinate" missing information. The MKBE framework by Pouya Pezeshkpour and Sameer Singh uses its multimodal embeddings to generate missing attributes, such as creating a text description for an entity that only has an image, or vice versa.

The Future: Collaborative Experts

The field is now moving toward "foundation" versions of these architectures. Fengtao Zhou and colleagues recently proposed MICE (Multimodal data Integration via Collaborative Experts), which scales this idea to 30 different cancer types. By using a "collaborative expert" architecture—where specialized encoders for pathology, genomics, and clinical reports work in tandem—they achieve high data efficiency, proving that a few well-encoded modalities are often more powerful than a mountain of poorly integrated data.

Papers in this version

Showing 60 of 167 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_modality_encoding_for_mult_df432e92,
  title = {Heterogeneous Modality Encoding for Multimodal Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-modality-encoding-for-multimodal-deep-learning/rdr_df432e92436a472e85d6663de0695304},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_modality_encoding_for_mult_df432e92_v1,
  title = {Heterogeneous Modality Encoding for Multimodal Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-modality-encoding-for-multimodal-deep-learning/rdr_df432e92436a472e85d6663de0695304/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}