LANISTR: Multimodal Learning from Structured and Unstructured Data

Sayna Ebrahimi, Sercan Ö. Arik, Yihe Dong, Tomas PfisterarXiv 2023 · DOI · arXiv

Abstract

LANISTR (LANguage, Image, and STRuctured data) is a novel multimodal framework designed to learn from a heterogeneous mix of unstructured data (text, images) and structured data (tables, time-series). By leveraging a Similarity-based Multimodal Masking (SimMMM) objective, LANISTR achieves state-of-the-art performance in low-resource settings and demonstrates high robustness to missing modalities, a common challenge in real-world datasets like healthcare and retail.

Introduction

While current foundation models excel at processing unstructured data—such as images and text—real-world applications in industries like healthcare and finance rely heavily on structured data (e.g., sensor readings, tabular databases). Existing multimodal models often treat structured data as an afterthought or struggle with "messy" datasets where modalities are frequently missing. LANISTR addresses these gaps by providing a unified framework for processing both structured and unstructured data, even when parallel data is scarce.

Methods

The LANISTR framework consists of two primary components: modality-specific encoders and a multimodal fusion encoder.

The LANISTR architecture utilizes specialized encoders for text, images, and structured data, followed by a multimodal fusion encoder.

Modality-Specific Encoders

To handle diverse data types effectively, LANISTR employs specialized architectures:

  • Text: A BERT-based encoder.
  • Images: A Vision Transformer (ViT).
  • Tabular Data: TabNet, specifically designed for tabular structures.
  • Time-Series: A Transformer-based encoder.

Similarity-based Multimodal Masking (SimMMM)

The core innovation of LANISTR is the Similarity-based Multimodal Masking (SimMMM) objective. Unlike standard reconstruction tasks, SimMMM creates two views of a multimodal sample: one original and one heavily masked (e.g., missing image patches or tabular columns). The model is trained to maximize the cosine similarity between the fused embeddings of these two views. This approach encourages the model to learn cross-modal relationships rather than simple feature memorization.

The SimMMM objective maximizes similarity between a complete multimodal sample and a masked version to learn robust cross-modal representations.

Datasets and Experimental Setup

LANISTR was evaluated on two large-scale, real-world datasets:

  • MIMIC-IV (Healthcare): Includes clinical time-series, medical notes, and chest X-rays. Approximately 35.7% of samples are missing at least one modality.
  • Amazon Product Review (Retail): Features product images, text reviews, and tabular metadata. This dataset presents an extreme challenge with a 99.8% missingness ratio in pretraining data.

Key Results

LANISTR demonstrates significant improvements over existing baselines, particularly in data-scarce scenarios:

  • Healthcare Performance: On MIMIC-IV, LANISTR achieved an 87.37% AUROC for in-hospital mortality prediction, outperforming the previous state-of-the-art (MedFuse) by over 9% while using only 0.1% of available labels.
  • Retail Performance: On Amazon Fashion reviews, the model achieved 75.15% accuracy, a 23% absolute boost over baselines when fine-tuned on just 0.01% of the data (512 samples).
  • Robustness and Efficiency: The SimMMM objective allows the model to effectively "fill in the blanks" for missing modalities. Furthermore, the model shows strong generalization, performing well on new categories (e.g., Beauty) after pretraining on others (e.g., Office Products).

Conclusion

LANISTR represents a major step toward practical AI for complex, real-world environments. By treating structured data as a first-class citizen and utilizing a robust masking objective, it overcomes the limitations of traditional multimodal models in handling incomplete and low-label datasets. Future work will explore extending the framework to audio and video modalities and developing automated methods for modality selection.

Summary updated

Abstract

Multimodal large-scale pretraining has shown impressive performance for unstructured data such as language and image. However, a prevalent real-world scenario involves structured data types, tabular and time-series, along with unstructured data. Such scenarios have been understudied. To bridge this gap, we propose LANISTR, an attention-based framework to learn from LANguage, Image, and STRuctured data. The core of LANISTR's methodology is rooted in \textit{masking-based} training applied across both unimodal and multimodal levels. In particular, we introduce a new similarity-based multimodal masking loss that enables it to learn cross-modal relations from large-scale multimodal data with missing modalities. On two real-world datasets, MIMIC-IV (from healthcare) and Amazon Product Review (from retail), LANISTR demonstrates remarkable improvements, 6.6\% (in AUROC) and 14\% (in accuracy) when fine-tuned with 0.1\% and 0.01\% of labeled data, respectively, compared to the state-of-the-art alternatives. Notably, these improvements are observed even with very high ratio of samples (35.7\% and 99.8\% respectively) not containing all modalities, underlining the robustness of LANISTR to practical missing modality challenge. Our code and models will be available at https://github.com/google-research/lanistr
BibTeX
@misc{ebrahimi2023lanistr,
  author = {Sayna Ebrahimi and Sercan Ö. Arik and Yihe Dong and Tomas Pfister},
  title = {{LANISTR: Multimodal Learning from Structured and Unstructured Data}},
  year = {2023},
  eprint = {2305.16556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.16556},
}
Figure