MuMIC - Multimodal Embedding for Multi-Label Image Classification with Tempered Sigmoid

Fengjun Wang, Sarai Mizrachi, Moran Beladev, Guy Nadav, Gil Amsalem, Karen Lastmann Assaraf, Hadas Harush BokerAAAI 2023 · DOI

TL;DR

This paper introduces MuMIC, a novel framework for multi-label image classification that leverages the power of multimodal learning (CLIP) and a custom "tempered sigmoid" loss function. MuMIC achieves state-of-the-art performance on a challenging real-world dataset from Booking.com, demonstrating improved robustness to noisy data and enabling effective zero-shot learning for unseen travel-related classes.

What Problem Does This Paper Address?

Multi-label image classification, where an image can belong to multiple categories simultaneously, is crucial for many applications. While recent advances in multimodal learning, particularly models like CLIP, have shown great promise in image-text representation, adapting them effectively to multi-label classification, especially in real-world scenarios with noisy data and the need for zero-shot capabilities, remains a challenge. Existing methods often struggle with label imbalance, regional feature extraction, and lack the ability to generalize to unseen classes without extensive retraining. This paper tackles these issues by proposing a framework that fine-tunes a pre-trained multimodal model with a specialized loss function designed for the complexities of multi-label classification.

Methods

The core of MuMIC is its adaptation of CLIP's multimodal architecture. The framework utilizes a Vision Transformer (ViT) as the image encoder and a Transformer as the text encoder, both initialized with CLIP's pre-trained weights. The key innovation lies in the tempered sigmoid based Binary Cross Entropy (BCE) loss function. This custom loss function is designed to be "hardness-aware," meaning it dynamically adjusts the penalty for misclassifications based on how "hard" a sample is. The "tempering" is achieved by introducing a temperature parameter into the sigmoid function, which allows for better control over the sensitivity to hard samples and improves robustness to noisy labels. This approach allows for efficient optimization of multi-label objectives and effective transfer learning from CLIP. Additionally, the paper explores enriching class context by using class descriptions alongside class names for label embeddings, and a simplified image preprocessing strategy (no cropping) that proved beneficial for their dataset.

Datasets

The primary dataset used in this paper is a newly constructed in-house dataset from Booking.com. This dataset comprises approximately 60,000 images with over 140,000 positive annotations across 120 defined image classes. The dataset is noted to be representative of real-world travel domain data and inherently contains noise. For evaluation and comparison, the authors also reference standard multi-label datasets like MS-COCO, although their primary experiments and results are on their custom Booking.com dataset.

Key Insights and Ideas

  • Tempered Sigmoid for Hardness-Awareness: The introduction of a temperature parameter in the sigmoid function for the BCE loss is a critical insight. It allows the model to be more forgiving of noisy labels while still effectively penalizing difficult misclassifications, leading to improved robustness.
  • No Cropping for Better Context: Surprisingly, for their specific domain (travel images), removing the standard image cropping preprocessing step (used by CLIP) and instead using resized full images significantly improved performance. This suggests that for certain domains, preserving the entire scene context is more important than focusing on central objects.
  • Zero-Shot Generalization in Travel Domain: MuMIC demonstrates strong zero-shot capabilities, particularly for unseen classes within the travel domain. This is a significant advantage for scalability and adapting to new trends without constant retraining.
  • Efficient Training: Leveraging CLIP's pre-trained weights and the tempered sigmoid loss, MuMIC achieves convergence in significantly fewer epochs (3) compared to a baseline ASL model (18 epochs), highlighting the efficiency of their approach.

Key Results

MuMIC achieves state-of-the-art performance on the Booking.com dataset, outperforming baselines like ASL and original CLIP. Key quantitative results include:

  • 85.6% GAP@10 and 83.8% GAP on all 120 classes.
  • 90.1% macro mAP across 32 majority classes.
  • Significantly improved performance on travel domain-related classes compared to CLIP.
  • Zero-shot prediction examples show MuMIC correctly identifying relevant labels (e.g., "internal museum," "mountain climbing") while assigning low scores to irrelevant ones (e.g., "shopping," "door"), indicating good generalization.

Qualitatively, t-SNE visualizations show that MuMIC embeddings lead to better differentiation among categories compared to original CLIP embeddings.

Limitations and Future Work

The authors acknowledge that even with their improvements, the dataset still contains noise, which is a common challenge in real-world problems. They suggest that refining class definitions, improving representations, reducing annotation noise, and assigning higher loss weights to specific classes could further enhance performance on low-performing labels. Future work also includes exploring few-shot learning for new classes using MuMIC embeddings.

Summary updated

BibTeX
@inproceedings{wang2023mumic,
  author = {Fengjun Wang and Sarai Mizrachi and Moran Beladev and Guy Nadav and Gil Amsalem and Karen Lastmann Assaraf and Hadas Harush Boker},
  title = {{MuMIC - Multimodal Embedding for Multi-Label Image Classification with Tempered Sigmoid}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26850},
}
Figure