Abstract
Contrastive Language-Image Pre-training (CLIP) has set new benchmarks in zero-shot learning but often struggles with fine-grained supervised classification on noisy, real-world data. We present CMA-CLIP, a framework that enhances CLIP by introducing sequence-wise and modality-wise attention mechanisms. These additions allow the model to capture fine-grained interactions between image patches and text tokens while dynamically weighting the importance of each modality. Our experiments on large-scale e-commerce and food datasets demonstrate significant performance gains over standard CLIP and other state-of-the-art multi-modal models.
Introduction
While OpenAI’s CLIP has revolutionized vision-language tasks, its application to supervised classification reveals several bottlenecks. First, CLIP primarily focuses on global features, lacking explicit modeling of local interactions—for instance, linking a "v-neck" token to specific pixels. Second, it treats image and text modalities as equally informative, which is often untrue in real-world scenarios like e-commerce, where product titles may be generic or images may be unrepresentative. Finally, downstream performance often hinges on extensive prompt engineering.
CMA-CLIP addresses these challenges by integrating two layers of attention into the CLIP backbone, making the model more robust to noise and better at capturing fine-grained cross-modal relationships.
Methodology: The CMA-CLIP Architecture
CMA-CLIP utilizes a CLIP (ViT-B/32) backbone and introduces three primary components to handle multi-modal classification tasks. The training follows a three-stage pipeline: a warm-up stage (freezing CLIP, training attention), an end-to-end stage (fine-tuning everything), and a tuning stage (freezing the backbone to optimize multi-task heads).
1. Sequence-wise Attention
This Transformer-based module facilitates communication between image patch embeddings and text token embeddings. By allowing these elements to interact, the resulting embeddings become context-aware. For example, the representation of a specific color in an image is strengthened by the presence of the corresponding color word in the text description.
2. Modality-wise Attention
To handle varying levels of informativeness between modalities, CMA-CLIP employs a "keyless attention" mechanism. This module learns to assign dynamic weights to image and text features based on their task relevance. If a text description is noisy or irrelevant for a specific attribute (like "color"), the model can learn to prioritize the visual information.
3. Multi-task Classification Heads
Unlike standard CLIP, CMA-CLIP supports simultaneous classification across multiple attributes. Task-specific Multi-Layer Perceptrons (MLPs) are attached to the output of the modality-wise attention layer, allowing the model to predict features like color, pattern, and style in a single pass.
Datasets and Evaluation
The model was evaluated on three diverse datasets characterized by varying levels of granularity and noise.
- MRWPA (Major Retail Website Product Attribute): A massive internal Amazon dataset of 5.8 million dress listings, focusing on attributes like Color, Pattern, and Style.
- Food101: A public dataset containing 101,000 images across 101 food categories.
- Fashion-Gen: A dataset of ~293,000 fashion images with professional descriptions, used for 121 sub-category classifications.
Key Insights and Results
CMA-CLIP consistently outperforms both fine-tuned CLIP and specialized state-of-the-art (SOTA) models.
Performance Gains
On the MRWPA dataset, CMA-CLIP achieved an average 11.9% improvement in recall (at 90% precision) compared to fine-tuned CLIP. In the Fashion-Gen benchmark, it reached 93.6% accuracy, surpassing the previous SOTA, KaleidoBERT, by 5.5%. Similarly, it achieved 93.1% accuracy on Food101, outperforming multi-modal baselines like MMBT.
Robustness to Noise
A key finding was the effectiveness of modality-wise attention. Ablation studies showed that removing this module led to a significant drop in recall (from 53.4% to 47.5% on MRWPA). The model demonstrates high resilience in practical application: when artificial noise was added to text titles, CMA-CLIP successfully ignored the irrelevant tokens, whereas models without modality-wise attention suffered performance degradation. Visualizations in the original research further confirm that the model successfully "looks" at the correct parts of an image when prompted by specific text tokens.
Conclusion and Future Directions
CMA-CLIP bridges the gap between the powerful global representations of two-stream models like CLIP and the fine-grained interaction capabilities of one-stream models. While highly effective for supervised classification, future work will explore training this architecture with contrastive loss on even larger datasets to enhance generalizability. Additionally, the authors plan to investigate the model's robustness against "noisy labels" (incorrect ground truth) and leverage semi-supervised learning techniques to utilize unlabeled data.
Reference: Liu, H., et al. (2021). CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification. arXiv:2112.03562v2 [cs.CV].
Summary updated