DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era

Chenwei Wu, David S. Restrepo, Chenwei Wu 0006, Constanza Vásquez-Venegas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M. LópezarXiv 2024 · DOI · arXiv

Abstract

In the era of big data, integrating diverse information sources—ranging from electronic health records to satellite imagery—remains a significant challenge due to high dimensionality and data heterogeneity. This paper introduces DF-DM (Data Fusion for Data Mining), a novel process model that bridges the gap between traditional data fusion frameworks and modern artificial intelligence. By utilizing "disentangled dense fusion" and foundation model embeddings, DF-DM provides a structured, resource-efficient approach to multimodal integration, demonstrating state-of-the-art performance in high-stakes domains like healthcare and social science.

Introduction

Modern decision-making environments are characterized by an abundance of data from disparate sources. However, current industry standards often struggle with three primary hurdles: the massive computational power required for raw high-dimensional data, the difficulty of finding a common mathematical language for heterogeneous sources (e.g., pixels vs. text), and the rigidity of existing frameworks like the military-originated JDL or DFIG models.

DF-DM addresses these issues by injecting the iterative, "closed-loop" philosophy of the CRISP-DM standard into a multi-level fusion architecture. This allows for a more flexible and reliable data mining process that can adapt to the complexities of real-world datasets.

Methods: The DF-DM Framework

The core of the DF-DM model is its ability to transform raw data into a manageable format while preserving essential information. The process is structured across several levels, integrating machine learning with human expertise.

The DF-DM framework integrates traditional data fusion levels with modern AI/ML techniques and human-in-the-loop decision making.

Key Technical Innovations

  • Foundation Model Embeddings: Rather than training on raw data, DF-DM utilizes pre-trained foundation models (such as DINO v2 for images and Llama 2 for text) to extract low-dimensional numerical vectors, or embeddings. This "Level 0" processing significantly reduces the computational barrier for downstream tasks.
  • Disentangled Dense Fusion: This transformer-based architecture separates "modality-common" features (shared information) from "modality-specific" features (unique information). This prevents the model from becoming redundant or ignoring less obvious but critical data streams.
  • Mutual Information (MI) Optimization: By employing a variational upper bound (vCLUB), the model minimizes redundancy between fused branches, ensuring that the learned features are as distinct and useful as possible.
  • Bias Assessment (Level 7): A dedicated level is included to explicitly identify and mitigate biases in the data, the model architecture, and the final decision-making process.

Experimental Results

The authors validated DF-DM across three complex use cases, consistently outperforming single-modality and standard dense fusion approaches:

  1. Diabetic Retinopathy (BRSET): Predicting retinopathy from fundus images and clinical metadata, the model achieved a Macro F1 score of 0.92.
  2. Domestic Violence Prediction: Combining satellite imagery, Google Trends, and census data, the model reached an R-squared of 0.854, showing a 4.2% improvement over standard fusion methods by eliminating redundant information.
  3. Radiology (MIMIC-CXR): In predicting diseases from chest X-rays and clinical notes, the model achieved a Macro AUC of 0.92.

Discussion and Future Work

DF-DM demonstrates that embeddings serve as a "great equalizer," allowing relatively simple linear models to achieve high performance when fed high-quality, low-dimensional data. However, the model's efficiency is contingent on the availability of high-quality pre-trained foundation models for specific domains.

Furthermore, while the inclusion of human-in-the-loop refinement (Level 5) is a significant strength, it introduces higher operational costs. Future research will focus on more efficient dimensionality reduction and refining human-model interaction to minimize resource requirements while maintaining the benefits of the iterative "loop" provided by the CRISP-DM integration.

Summary updated

Abstract

In the big data era, integrating diverse data modalities poses significant challenges, particularly in complex fields like healthcare. This paper introduces a new process model for multimodal Data Fusion for Data Mining, integrating embeddings and the Cross-Industry Standard Process for Data Mining with the existing Data Fusion Information Group model. Our model aims to decrease computational costs, complexity, and bias while improving efficiency and reliability. We also propose "disentangled dense fusion", a novel embedding fusion method designed to optimize mutual information and facilitate dense inter-modality feature interaction, thereby minimizing redundant information. We demonstrate the model's efficacy through three use cases: predicting diabetic retinopathy using retinal images and patient metadata, domestic violence prediction employing satellite imagery, internet, and census data, and identifying clinical and demographic features from radiography images and clinical notes. The model achieved a Macro F1 score of 0.92 in diabetic retinopathy prediction, an R-squared of 0.854 and sMAPE of 24.868 in domestic violence prediction, and a macro AUC of 0.92 and 0.99 for disease prediction and sex classification, respectively, in radiological analysis. These results underscore the Data Fusion for Data Mining model's potential to significantly impact multimodal data processing, promoting its adoption in diverse, resource-constrained settings.
BibTeX
@misc{wu2024df,
  author = {Chenwei Wu and David S. Restrepo and Chenwei Wu and Constanza Vásquez-Venegas and Luis Filipe Nakayama and Leo Anthony Celi and Diego M. López},
  title = {{DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era}},
  year = {2024},
  eprint = {2404.12278},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.12278},
}
Figure