Abstract
In the era of big data, integrating diverse information sources—ranging from electronic health records to satellite imagery—remains a significant challenge due to high dimensionality and data heterogeneity. This paper introduces DF-DM (Data Fusion for Data Mining), a novel process model that bridges the gap between traditional data fusion frameworks and modern artificial intelligence. By utilizing "disentangled dense fusion" and foundation model embeddings, DF-DM provides a structured, resource-efficient approach to multimodal integration, demonstrating state-of-the-art performance in high-stakes domains like healthcare and social science.
Introduction
Modern decision-making environments are characterized by an abundance of data from disparate sources. However, current industry standards often struggle with three primary hurdles: the massive computational power required for raw high-dimensional data, the difficulty of finding a common mathematical language for heterogeneous sources (e.g., pixels vs. text), and the rigidity of existing frameworks like the military-originated JDL or DFIG models.
DF-DM addresses these issues by injecting the iterative, "closed-loop" philosophy of the CRISP-DM standard into a multi-level fusion architecture. This allows for a more flexible and reliable data mining process that can adapt to the complexities of real-world datasets.
Methods: The DF-DM Framework
The core of the DF-DM model is its ability to transform raw data into a manageable format while preserving essential information. The process is structured across several levels, integrating machine learning with human expertise.
Key Technical Innovations
- Foundation Model Embeddings: Rather than training on raw data, DF-DM utilizes pre-trained foundation models (such as DINO v2 for images and Llama 2 for text) to extract low-dimensional numerical vectors, or embeddings. This "Level 0" processing significantly reduces the computational barrier for downstream tasks.
- Disentangled Dense Fusion: This transformer-based architecture separates "modality-common" features (shared information) from "modality-specific" features (unique information). This prevents the model from becoming redundant or ignoring less obvious but critical data streams.
- Mutual Information (MI) Optimization: By employing a variational upper bound (vCLUB), the model minimizes redundancy between fused branches, ensuring that the learned features are as distinct and useful as possible.
- Bias Assessment (Level 7): A dedicated level is included to explicitly identify and mitigate biases in the data, the model architecture, and the final decision-making process.
Experimental Results
The authors validated DF-DM across three complex use cases, consistently outperforming single-modality and standard dense fusion approaches:
- Diabetic Retinopathy (BRSET): Predicting retinopathy from fundus images and clinical metadata, the model achieved a Macro F1 score of 0.92.
- Domestic Violence Prediction: Combining satellite imagery, Google Trends, and census data, the model reached an R-squared of 0.854, showing a 4.2% improvement over standard fusion methods by eliminating redundant information.
- Radiology (MIMIC-CXR): In predicting diseases from chest X-rays and clinical notes, the model achieved a Macro AUC of 0.92.
Discussion and Future Work
DF-DM demonstrates that embeddings serve as a "great equalizer," allowing relatively simple linear models to achieve high performance when fed high-quality, low-dimensional data. However, the model's efficiency is contingent on the availability of high-quality pre-trained foundation models for specific domains.
Furthermore, while the inclusion of human-in-the-loop refinement (Level 5) is a significant strength, it introduces higher operational costs. Future research will focus on more efficient dimensionality reduction and refining human-model interaction to minimize resource requirements while maintaining the benefits of the iterative "loop" provided by the CRISP-DM integration.
Summary updated