Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust

Vera Pavlova, Mohammed MakhloufEMNLP 2024

TL;DR

This paper introduces an efficient, multilingual, non-profit Information Retrieval (IR) system for the Islamic domain, tackling the challenges of limited data and resource constraints. They achieve this by developing a lightweight, domain-adapted multilingual language model (MLLM) and leveraging Rust's multiprocessing capabilities for fast, CPU-based semantic search.

What Problem Does This Paper Address?

The widespread success of AI in Natural Language Processing (NLP), particularly with Large Language Models (LLMs), has largely benefited commercially driven domains. However, areas with less commercial interest, such as religious and heritage corpora, often lag behind in adopting these advanced technologies. The Islamic domain, rich in cultural and scholarly texts, presents a significant challenge for information retrieval due to the difficulty of finding unified, AI-powered search resources. Key issues include:

  • Data Scarcity: Multilingual domain-specific corpora are hard to come by, especially for less-resourced languages.
  • Model Size: State-of-the-art multilingual models are often very large (over 1GB), making them difficult and expensive to deploy, particularly on resource-constrained, non-profit infrastructure.
  • Deployment Costs: Relying on GPU acceleration for fast inference is often prohibitive for non-profit organizations with limited budgets.

Methods

The authors propose a multi-pronged approach to build an efficient and accessible IR system:

  • Domain Adaptation: To address the "domain shift" problem where general-domain LLMs perform poorly on specialized texts, they employ a continued pre-training strategy. This involves further training a multilingual LLM (specifically, XLM-RBase) on a custom-built Islamic domain corpus.
  • Language Reduction: To combat the large model size, they implement a language reduction technique. This involves training a new tokenizer that focuses only on the necessary languages (English, Russian, Arabic, Urdu) and then selectively copying encoder weights from the original XLM-RBase model. This results in a significantly smaller model (XLM-R4) with fewer parameters.
  • Domain-Specific Vocabulary: To further improve domain adaptation, they introduce new domain-specific vocabulary into the XLM-R4 model, creating the XLM-R4-ID model.
  • Efficient Deployment with Rust: To overcome the cost barrier of GPUs, they leverage the multiprocessing capabilities of Rust. Rust is chosen for its safety, efficiency, and lightweight memory footprint, allowing for fast, CPU-based semantic search without the overhead of a Python interpreter or garbage collection. They implement a system where corpus embeddings are chunked and distributed across multiple worker threads, enabling parallel processing of search queries.

Datasets

  • Training/Domain Adaptation:
    • A multilingual variant of the C4 corpus was compiled for the languages of interest (English, Russian, Arabic, and Urdu).
  • The Open Islamicate Texts Initiative (OpenITI) corpus (approximately 1 billion words in Classical Arabic) was used, with a random subset of 250 million words selected for domain adaptation.
  • The MS MARCO IR dataset (English) and its Arabic machine-translated version were used for training retrieval models.
  • Evaluation:
    • The XNLI dataset was used for evaluating cross-lingual transfer performance of the language reduction technique.
  • The Arabic QRCD (Qur'anic Reading Comprehension Dataset) and its verified translations into English, Russian, and Urdu were used for in-domain IR evaluation.
  • The Holy Quran text (Arabic) and its translations (English, Russian, Urdu) were used as the retrieval corpus.
  • The Hadith Question-Answer pairs (HAQA) dataset was used for measuring retrieval time and recall on a larger corpus.

Key Insights and Ideas

  • Domain Adaptation is Crucial: Even with a relatively small domain-specific corpus, continued pre-training and the introduction of domain-specific vocabulary significantly improve performance on specialized IR tasks, outperforming general-domain models.
  • Language Reduction is Effective: Trimming the embedding matrix of MLLMs by removing unnecessary languages can drastically reduce model size while retaining high performance, making deployment more feasible.
  • Rust for Efficient AI: Rust's multiprocessing capabilities offer a viable and efficient alternative to GPU-accelerated systems for semantic search, especially in resource-constrained environments, without sacrificing performance.
  • Arabic as a Strong Transfer Language: For the Islamic domain, Arabic appears to be a particularly effective transfer language, as evidenced by the performance of models trained with Arabic data.

Key Results

  • The developed XLM-R4-ID model (481MB, 119M parameters) significantly outperforms general-domain multilingual and monolingual models on the in-domain IR dataset.
  • The XLM-R4-ID model shows a slight drop in performance compared to the original XLM-RBase on the XNLI dataset but still outperforms other baselines like mBERT and DistilmBERT.
  • The Rust-based semantic search system achieves a speedup of up to 4.9x compared to Python baselines (e.g., HNSW) without compromising recall (maintaining 100% recall).
  • The Rust implementation demonstrates a speedup of 2.6x even with a single worker, with further improvements achievable by adding more workers.

Limitations and Future Work

  • Corpus Size for Evaluation: The evaluation of retrieval time and recall was performed on a smaller retrieval corpus (approx. 50k passages) than the full dataset (over 150k passages).
  • Future Work: The paper implicitly suggests that further exploration of language reduction techniques and domain adaptation strategies could lead to even more efficient and performant models. The successful deployment of a Rust-based system also opens avenues for exploring other computationally intensive AI tasks in low-resource settings using Rust.

Summary updated

BibTeX
@inproceedings{pavlova2024building,
  author = {Vera Pavlova and Mohammed Makhlouf},
  title = {{Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure