Abstract
This paper focuses on multimodal alignment within the realm of Artificial Intelligence, particularly in text and image modalities. The semantic gap between the textual and visual modality poses a discrepancy problem towards the effectiveness of multi-modalities fusion. Therefore, we introduce Text-Image Joint Embedding Predictive Architecture (TI-JEPA), an innovative pre-training strategy that leverages energy-based model (EBM) framework to capture complex cross-modal relationships. TI-JEPA combines the flexibility of EBM in self-supervised learning to facilitate the compatibility between textual and visual elements. Through extensive experiments across multiple benchmarks, we demonstrate that TI-JEPA achieves state-of-the-art performance on multimodal sentiment analysis task (and potentially on a wide range of multimodal-based tasks, such as Visual Question Answering), outperforming existing pre-training methodologies. Our findings highlight the potential of using energy-based framework in advancing multimodal fusion and suggest significant improvements for downstream applications.
Introduction
In the field of multimodal Artificial Intelligence, achieving effective "alignment" between disparate data types remains a primary challenge. Specifically, a significant semantic gap exists between visual and textual modalities: images are represented in high-dimensional, continuous spaces, whereas text is discrete and symbolic. Traditional fusion techniques, such as simple concatenation or contrastive learning (e.g., CLIP), often struggle to capture the intricate, latent relationships between specific image regions and their corresponding textual descriptions.
To address this, we propose TI-JEPA (Text-Image Joint Embedding Predictive Architecture). Unlike generative models that focus on pixel-level reconstruction, TI-JEPA utilizes a non-generative, predictive approach. It aims to bridge the semantic gap by matching internal representations, leveraging an energy-based framework to facilitate deep compatibility between textual and visual elements.
Methodology
TI-JEPA extends the principles of the Image-based Joint-Embedding Predictive Architecture (I-JEPA) into the multimodal domain. The core innovation lies in using one modality as the context to predict the representation of another.
Architectural Components
The architecture consists of several key modules:
- Dual Encoders: The system employs a pre-trained Vision Transformer (ViT-H) for image processing and a transformer-based text encoder (gte-base-en-v1.5).
- Cross-Attention Mechanism: Acting as the "bridge," text-to-image (t2i) cross-attention blocks align encoded text features with visual features.
- Predictive Masking: During training, specific patches of an image are masked. The model is tasked with predicting the embeddings of these missing patches using the remaining image context and the provided text description.
- Energy-Based Objective: The model minimizes the $L_2$ distance (energy) between the predicted representation and the actual representation of the target masked patches.
Preventing Energy Collapse
A common challenge in training energy-based models is "energy collapse," where the model learns to output a constant representation regardless of the input. To mitigate this, TI-JEPA utilizes strategic freezing. By freezing the high-quality, pre-trained image and text encoders and only training the cross-attention and predictor modules, the model maintains representational diversity and bypasses the instability of training EBMs from scratch.
Experimental Setup
The model was pre-trained on the Microsoft COCO 2017 dataset, utilizing over 118,000 image-text pairs. For evaluation, we focused on multimodal sentiment analysis using the MVSA-Single and MVSA-Multi datasets. These datasets contain image-text pairs from Twitter, which were rigorously pre-processed to ensure emotional consistency between the two modalities.
We developed three variants of the architecture—Small, Medium, and Large—to evaluate the impact of model scale on performance. As shown in the configuration table, these range from 39M to 131M parameters.
Results and Discussion
TI-JEPA demonstrates superior performance in capturing cross-modal relationships compared to established benchmarks.
- State-of-the-Art Accuracy: The TI-JEPA-Large model achieved an accuracy of 76.75% on MVSA-Single and 77.55% on MVSA-Multi.
- Comparison with CLIP: TI-JEPA outperformed the CLIP-CA-CG model, which scored 75.25% and 76.05% on the respective datasets. This indicates that the joint embedding predictive approach is more effective at sentiment alignment than standard contrastive methods.
- Efficiency: Notably, the "Small" version of TI-JEPA (39M parameters) provided competitive results, suggesting that the architecture is inherently efficient at learning high-level semantics without requiring massive parameter counts.
Conclusion
TI-JEPA introduces a robust framework for multimodal alignment by focusing on representations rather than pixels. By successfully extending the JEPA paradigm to text-image pairs, we have shown that energy-based models can effectively bridge the semantic gap.
Future research will explore expanding the architecture to include additional modalities such as audio and video. Furthermore, we aim to investigate fully end-to-end pre-training pipelines to allow the base encoders to adapt more specifically to multimodal nuances, potentially enhancing performance on complex tasks like Visual Question Answering (VQA).
Summary updated