CMA-CLIP: Cross-Modality Attention Clip for Text-Image Classification

Jinmiao Fu, Shaoyuan Xu, Huidong Liu, Yang Liu, Ning Xie, Chien-Chih Wang, Jia Liu, Yi Sun, Bryan Wang

Abstract

Contrastive Language-Image Pre-training (CLIP) has set new benchmarks in zero-shot learning but often struggles with fine-grained supervised classification on noisy, real-world data. We present CMA-CLIP, a framework that enhances CLIP by introducing sequence-wise and modality-wise attention mechanisms. These additions allow the model to capture fine-grained interactions between image patches and text tokens while dynamically weighting the importance of each modality. Our experiments on large-scale e-commerce and food datasets demonstrate significant performance gains over standard CLIP and other state-of-the-art multi-modal models.

Introduction

While OpenAI’s CLIP has revolutionized vision-language tasks, its application to supervised classification reveals several bottlenecks. First, CLIP primarily focuses on global features, lacking explicit modeling of local interactions—for instance, linking a "v-neck" token to specific pixels. Second, it treats image and text modalities as equally informative, which is often untrue in real-world scenarios like e-commerce, where product titles may be generic or images may be unrepresentative. Finally, downstream performance often hinges on extensive prompt engineering.

CMA-CLIP addresses these challenges by integrating two layers of attention into the CLIP backbone, making the model more robust to noise and better at capturing fine-grained cross-modal relationships.

Methodology: The CMA-CLIP Architecture

CMA-CLIP utilizes a CLIP (ViT-B/32) backbone and introduces three primary components to handle multi-modal classification tasks. The training follows a three-stage pipeline: a warm-up stage (freezing CLIP, training attention), an end-to-end stage (fine-tuning everything), and a tuning stage (freezing the backbone to optimize multi-task heads).

The CMA-CLIP architecture featuring sequence-wise and modality-wise attention modules.

1. Sequence-wise Attention

This Transformer-based module facilitates communication between image patch embeddings and text token embeddings. By allowing these elements to interact, the resulting embeddings become context-aware. For example, the representation of a specific color in an image is strengthened by the presence of the corresponding color word in the text description.

2. Modality-wise Attention

To handle varying levels of informativeness between modalities, CMA-CLIP employs a "keyless attention" mechanism. This module learns to assign dynamic weights to image and text features based on their task relevance. If a text description is noisy or irrelevant for a specific attribute (like "color"), the model can learn to prioritize the visual information.

3. Multi-task Classification Heads

Unlike standard CLIP, CMA-CLIP supports simultaneous classification across multiple attributes. Task-specific Multi-Layer Perceptrons (MLPs) are attached to the output of the modality-wise attention layer, allowing the model to predict features like color, pattern, and style in a single pass.

Datasets and Evaluation

The model was evaluated on three diverse datasets characterized by varying levels of granularity and noise.

Examples from the MRWPA, Food101, and Fashion-Gen datasets used for evaluation.
  • MRWPA (Major Retail Website Product Attribute): A massive internal Amazon dataset of 5.8 million dress listings, focusing on attributes like Color, Pattern, and Style.
  • Food101: A public dataset containing 101,000 images across 101 food categories.
  • Fashion-Gen: A dataset of ~293,000 fashion images with professional descriptions, used for 121 sub-category classifications.

Key Insights and Results

CMA-CLIP consistently outperforms both fine-tuned CLIP and specialized state-of-the-art (SOTA) models.

Performance Gains

On the MRWPA dataset, CMA-CLIP achieved an average 11.9% improvement in recall (at 90% precision) compared to fine-tuned CLIP. In the Fashion-Gen benchmark, it reached 93.6% accuracy, surpassing the previous SOTA, KaleidoBERT, by 5.5%. Similarly, it achieved 93.1% accuracy on Food101, outperforming multi-modal baselines like MMBT.

Robustness to Noise

A key finding was the effectiveness of modality-wise attention. Ablation studies showed that removing this module led to a significant drop in recall (from 53.4% to 47.5% on MRWPA). The model demonstrates high resilience in practical application: when artificial noise was added to text titles, CMA-CLIP successfully ignored the irrelevant tokens, whereas models without modality-wise attention suffered performance degradation. Visualizations in the original research further confirm that the model successfully "looks" at the correct parts of an image when prompted by specific text tokens.

Conclusion and Future Directions

CMA-CLIP bridges the gap between the powerful global representations of two-stream models like CLIP and the fine-grained interaction capabilities of one-stream models. While highly effective for supervised classification, future work will explore training this architecture with contrastive loss on even larger datasets to enhance generalizability. Additionally, the authors plan to investigate the model's robustness against "noisy labels" (incorrect ground truth) and leverage semi-supervised learning techniques to utilize unlabeled data.

Reference: Liu, H., et al. (2021). CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification. arXiv:2112.03562v2 [cs.CV].

Summary updated

Abstract

Multi-modal learning with both text and images benefits multiple applications, such as attribute extraction for e-commerce products. In this paper, we propose Cross-Modality Attention Contrastive Language-Image Pre-training (CMA-CLIP), a new multi-modal architecture to jointly learn the fine-grained inter-modality relationship. It fuses CLIP with a sequence-wise attention module and a modality-wise attention module. The network uses CLIP to bridge the inter-modality gap at the global level, and uses the sequence-wise attention module to capture the fine-grained alignment between text and images. Besides, it leverages a modality-wise attention module to learn the relevance of each modality to downstream tasks, making the network robust against irrelevant modalities. CMA-CLIP outperforms the state-of-the-art method on Fashion-Gen by 5.5% in accuracy, achieves competitive performance on Food101 and performance on par with the state-of-the-art method on MM-IMDb. We also demonstrate CMA-CLIP’s robustness against irrelevant modalities on an Amazon dataset for the task of product attribute extraction.
BibTeX
@inproceedings{fu2022cma,
  author = {Jinmiao Fu and Shaoyuan Xu and Huidong Liu and Yang Liu and Ning Xie and Chien-Chih Wang and Jia Liu and Yi Sun and Bryan Wang},
  title = {{CMA-CLIP: Cross-Modality Attention Clip for Text-Image Classification}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=fNhwX0TEgA},
}

Loading related content...

Figure