Abstract
Traditional multimodal learning models often rely on a static set of tabular attributes, making it difficult to incorporate new data discovered after training without costly retraining. We present MIITA, a framework that enables pre-trained multimodal models to integrate "incremental" tabular attributes during the inference stage. By utilizing a specialized disentangled encoder and unsupervised alignment losses, MIITA extracts novel information from new data fields while preserving existing knowledge. Experimental results across five diverse datasets demonstrate that MIITA consistently outperforms existing static models and even improves performance in fully supervised settings.
Introduction
In dynamic fields such as healthcare and finance, the data landscape is constantly evolving. A model trained to predict Alzheimer’s disease using MRI scans and a specific set of clinical markers may suddenly need to incorporate a newly discovered protein biomarker. In a traditional AI workflow, incorporating this new data field typically requires retraining the entire model from scratch—a process often hindered by privacy constraints or the inability to re-access historical patient data.
Most current multimodal learning methods assume a static environment where the number of tabular columns remains fixed. Tabular data is inherently heterogeneous; each column possesses its own scale and distribution. Simply appending a new column to a pre-trained encoder often leads to performance degradation. The challenge lies in extracting complementary information from these incremental attributes without requiring new ground-truth labels or a full system overhaul.
Methods
The MIITA framework addresses the "static table" problem by keeping the original visual and tabular encoders frozen to preserve existing knowledge. To process new data, it introduces a Disentangled Representation Learning (DRL) Encoder.
Technical Innovations
- DRL Encoder: Based on a $\beta$-Variational Autoencoder ($\beta$-VAE), this encoder disentangles the complex features of new attributes into independent, interpretable dimensions (e.g., spatial, clinical, or genetic factors).
- Incremental Disentangled Loss: This loss ensures that new features are not redundant. It utilizes KL-divergence to push the new representation away from the old one and employs covariance regularization to penalize linear correlations.
- Visual Consistency Loss: This mechanism aligns the combined tabular representation with the image representation. It ensures that the incremental attributes provide complementary value by forcing the new multimodal embedding to be closer to the visual representation than the original tabular data was alone.
- Unsupervised Adaptation: Since ground-truth labels for new data are often unavailable at inference time, MIITA employs pseudo-labels generated by the original model, filtering for high-confidence predictions to provide a soft supervision signal.
Experimental Evaluation
The authors evaluated MIITA across five public datasets: ADNI (Medical), Data Visual Marketing (Automotive), Pokemon Primary Type (Gaming), Hearthstone (Gaming), and CS:GO (Gaming). In each scenario, specific columns were withheld during training and re-introduced only during the inference phase to simulate real-world data evolution.
Key Results
MIITA consistently achieved state-of-the-art (SOTA) results, ranking first across all tested datasets. For instance, on the ADNI Alzheimer's diagnostic task, MIITA reached an accuracy of 0.814, significantly outperforming the standard multimodal baseline (MMCL) at 0.773.
As shown in the comparative results, baseline models like XGBoost and ViT often fail when critical information is missing from their initial training set. MIITA successfully leverages incremental attributes—such as a car's price or a Pokemon's ability—to correct these predictions during inference.
Discussion and Insights
A key finding of the study is the value of "weakly correlated" attributes. If a tabular attribute is highly correlated with the image, the visual encoder likely already captured that information. Attributes weakly related to the image often provide the most significant performance boosts because they offer truly novel information. Furthermore, MIITA can be used as a progressive training style; even in fully supervised settings, integrating features incrementally led to better results than training on all data simultaneously.
Conclusion
MIITA provides a robust solution for deploying multimodal models in dynamic environments. By allowing for the seamless integration of new tabular data without retraining, it bridges the gap between static model architectures and the evolving nature of real-world data. Future work will explore extending these principles to other modalities, such as audio and time-series data, and developing foundation models natively designed for dynamic input dimensions.
Summary updated