AraCLIP: Cross-Lingual Learning for Effective Arabic Image Retrieval

Muhammad Al-Barham, Imad Afyouni, Khalid Almubarak, Ashraf Elnagar, Ayad Turky, Ibrahim HashemACL 2024

TL;DR

This paper introduces AraCLIP, a novel model for Arabic text-to-image retrieval that leverages knowledge distillation from English CLIP to effectively understand Arabic language nuances. AraCLIP significantly outperforms existing multilingual models, demonstrating a substantial advancement in cross-lingual image retrieval for the Arabic language.

What Problem Does This Paper Address?

The field of text-to-image retrieval has seen significant progress, but most existing models are heavily focused on English. This leaves a large population of Arabic speakers underserved, despite the richness and complexity of the Arabic language, including its morphology, grammar, and diverse dialects. Developing effective text-to-image retrieval for Arabic requires models that can truly grasp these linguistic intricacies, a challenge that current multilingual models may not fully address.

Methods

The core of AraCLIP's approach is Knowledge Distillation, specifically a "teacher learning" strategy. Instead of training a model from scratch, they leverage a pre-trained English CLIP model as the "teacher" and an Arabic model (Arabert) as the "student." The process involves:

  1. English CLIP Inference: The English text encoder and image encoder are used to generate embeddings for English captions.
  2. Arabic Model Training: An Arabic text encoder is trained to produce embeddings that closely match those generated by the English teacher model. This is achieved by translating English captions to Arabic and then minimizing the Mean Squared Error (MSE) between the resulting embeddings of the English and Arabic text encoders. Crucially, this method avoids direct image involvement in the training of the Arabic text encoder, significantly reducing computational resources compared to traditional contrastive learning.
  3. Evaluation: The trained Arabic text encoder is then paired with the original CLIP image encoder to evaluate its performance on Arabic text-to-image retrieval tasks.

Datasets

The paper introduces several new Arabic datasets and utilizes existing ones:

  • Translated and Cleaned Datasets:
    • A translated and cleaned version of the CC3M+CC12M+SBU dataset (filtered by ViT-L), containing approximately 12.5 million caption-image pairs.
  • A translated version of MS COCO, with approximately 123,280 caption-image pairs.
  • A testing dataset, Arabic XTD-11, with about 1,000 caption-image pairs.
  • Evaluation Datasets:
    • Arabic XTD-11
    • Arabic Flicker 8k

The paper details extensive data cleaning processes, including removing duplicate text, text with justifications, English text/symbols, and extraneous characters like Harakat and Tatweel.

Key Insights and Ideas

  • Knowledge Distillation for Arabic: The paper demonstrates that knowledge distillation is an effective method for transferring cross-modal understanding from a high-resource language (English) to a low-resource language (Arabic) for image retrieval.
  • Arabic Language Nuances: AraCLIP is specifically designed to capture the unique terms, cultural nuances, and contextual constructs of the Arabic language, which is a significant advantage over generic multilingual models.
  • Efficiency of Teacher Learning: By using translated captions and minimizing MSE between embeddings, the authors bypass the need for direct image-text alignment during the Arabic text encoder's training, making the process computationally more efficient than traditional contrastive learning.
  • Base Models Outperform Large Models: Interestingly, the "base" variants of AraCLIP models (Arabert-v2-base-ViT-B-16-SigLIP-512-2M) showed better performance than the "large" variants (Arabert-v2-large-ViT-B-16-plus-240-2M) on the tested datasets, suggesting that increased model size doesn't always translate to better performance for this specific task.

Key Results

AraCLIP demonstrates significant improvements over existing multilingual models:

  • Quantitative Evaluation:
    • AraCLIP models consistently outperformed the mCLIP model (XLM-Roberta-Large-Vit-B-16Plus) across both Arabic XTD-11 and Arabic Flickr8k datasets.
  • Improvements were observed in Mean Reciprocal Rank (MRR) and Recall metrics, with AraCLIP achieving approximately 10% higher scores in MRR@1, MRR@5, and MRR@10.
  • For example, on the Arabic XTD-11 dataset, AraCLIP models achieved MRR@1 scores above 0.66, indicating a higher likelihood of returning the most relevant result.
  • Qualitative Evaluation:
    • In complex sentence categories, AraCLIP was better at retrieving relevant images, even capturing subtle details like "sail" in a "man surfing" query where mCLIP failed.
  • In "performance under noise" scenarios (e.g., "A dog attacks a cat while the cat is under a wooden bench"), AraCLIP was able to capture the correct image as the highest similarity result, whereas mCLIP struggled initially but could retrieve related images later.

Limitations and Future Work

The authors acknowledge that their model still requires further improvements. Promising directions for future work include:

  • Translation Quality: Further research into improving the quality of machine translation for Arabic captions.
  • Model Selection: Exploring different Arabic text encoders and visual transformers.
  • Evaluation Aspects: Developing more comprehensive and nuanced evaluation methodologies for Arabic text-to-image retrieval.
  • Expanding Datasets: Creating larger and more diverse Arabic datasets for training and evaluation.

Summary updated

BibTeX
@inproceedings{albarham2024araclip,
  author = {Muhammad Al-Barham and Imad Afyouni and Khalid Almubarak and Ashraf Elnagar and Ayad Turky and Ibrahim Hashem},
  title = {{AraCLIP: Cross-Lingual Learning for Effective Arabic Image Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure