Abstract
LANISTR (LANguage, Image, and STRuctured data) is a novel multimodal framework designed to learn from a heterogeneous mix of unstructured data (text, images) and structured data (tables, time-series). By leveraging a Similarity-based Multimodal Masking (SimMMM) objective, LANISTR achieves state-of-the-art performance in low-resource settings and demonstrates high robustness to missing modalities, a common challenge in real-world datasets like healthcare and retail.
Introduction
While current foundation models excel at processing unstructured data—such as images and text—real-world applications in industries like healthcare and finance rely heavily on structured data (e.g., sensor readings, tabular databases). Existing multimodal models often treat structured data as an afterthought or struggle with "messy" datasets where modalities are frequently missing. LANISTR addresses these gaps by providing a unified framework for processing both structured and unstructured data, even when parallel data is scarce.
Methods
The LANISTR framework consists of two primary components: modality-specific encoders and a multimodal fusion encoder.
Modality-Specific Encoders
To handle diverse data types effectively, LANISTR employs specialized architectures:
- Text: A BERT-based encoder.
- Images: A Vision Transformer (ViT).
- Tabular Data: TabNet, specifically designed for tabular structures.
- Time-Series: A Transformer-based encoder.
Similarity-based Multimodal Masking (SimMMM)
The core innovation of LANISTR is the Similarity-based Multimodal Masking (SimMMM) objective. Unlike standard reconstruction tasks, SimMMM creates two views of a multimodal sample: one original and one heavily masked (e.g., missing image patches or tabular columns). The model is trained to maximize the cosine similarity between the fused embeddings of these two views. This approach encourages the model to learn cross-modal relationships rather than simple feature memorization.
Datasets and Experimental Setup
LANISTR was evaluated on two large-scale, real-world datasets:
- MIMIC-IV (Healthcare): Includes clinical time-series, medical notes, and chest X-rays. Approximately 35.7% of samples are missing at least one modality.
- Amazon Product Review (Retail): Features product images, text reviews, and tabular metadata. This dataset presents an extreme challenge with a 99.8% missingness ratio in pretraining data.
Key Results
LANISTR demonstrates significant improvements over existing baselines, particularly in data-scarce scenarios:
- Healthcare Performance: On MIMIC-IV, LANISTR achieved an 87.37% AUROC for in-hospital mortality prediction, outperforming the previous state-of-the-art (MedFuse) by over 9% while using only 0.1% of available labels.
- Retail Performance: On Amazon Fashion reviews, the model achieved 75.15% accuracy, a 23% absolute boost over baselines when fine-tuned on just 0.01% of the data (512 samples).
- Robustness and Efficiency: The SimMMM objective allows the model to effectively "fill in the blanks" for missing modalities. Furthermore, the model shows strong generalization, performing well on new categories (e.g., Beauty) after pretraining on others (e.g., Office Products).
Conclusion
LANISTR represents a major step toward practical AI for complex, real-world environments. By treating structured data as a first-class citizen and utilizing a robust masking objective, it overcomes the limitations of traditional multimodal models in handling incomplete and low-label datasets. Future work will explore extending the framework to audio and video modalities and developing automated methods for modality selection.
Summary updated