Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios

Vera Pavlova, Mohammed MakhloufEMNLP 2025

TL;DR

This paper tackles the challenge of deploying efficient multilingual information retrieval (MLIR) systems in real-world scenarios, particularly for the Islamic domain. They propose a novel "mixed training" approach for a lightweight, domain-adapted model that achieves competitive performance across various retrieval tasks, making advanced MLIR more accessible and cost-effective.

What Problem Does This Paper Address?

The field of Multilingual Information Retrieval (MLIR) has seen significant progress, largely driven by large language models (LLMs). However, a persistent gap exists between academic research and practical deployment. Many MLIR studies focus on isolated settings or benchmark datasets, failing to address the complexities of real-world applications that often require a single system to handle diverse search scenarios (monolingual, cross-lingual, and multilingual). This is particularly true for specialized domains like religious texts, where nuanced understanding and accessibility across languages are crucial. The authors aim to bridge this gap by developing an efficient and versatile MLIR system for the Islamic domain, specifically focusing on retrieving passages from the Holy Quran.

Methods

The core of the paper lies in developing a lightweight, domain-specific multilingual large language model (MLLM) and exploring various training strategies.

  • Model Reduction: They start with the XLM-RBase model and apply language reduction techniques to create a smaller, more efficient XLM-R4 model (reduced from 1.1 GB to 481 MB) by keeping only the necessary language tokens.
  • Domain Adaptation: To improve performance on Islamic texts, they perform a short round of pre-training on a curated corpus of 100 million words, combining Arabic texts from the Open Islamicate Texts Initiative with English, Russian, and Urdu texts (Tafseer and Hadith). This results in the XLM-R4-ID model.
  • Training Approaches: Eleven retrieval models are trained using four distinct approaches:
  • Monolingual Training: Queries and passages are in the same language.
  • Cross-lingual Training: Queries are in one language, and passages are in another.
  • Mixed Approach (Novel): This approach combines monolingual and cross-lingual training by constructing training data in three ways: monolingual queries with bilingual collections, bilingual queries with monolingual collections, and bilingual queries with bilingual collections. The authors hypothesize this enhances training example diversity and cross-lingual interaction.
  • Translate-train-all Approach: This involves training on multiple translations of the dataset simultaneously.
  • Retrieval Framework: They employ a dense retrieval approach using the sentence transformer framework, which adds a pooling layer on top of LLM embeddings to produce fixed-sized sentence embeddings. A contrastive learning loss function is used to create an embedding space that brings related queries and passages closer.

Datasets

  • Training:
    • MS MARCO: A large-scale English-language dataset used for training and evaluating dense retrieval models. Machine-translated variants for 13 languages were used, and MS MARCO was translated into Urdu for this study.
  • Evaluation:
    • QRCD (Qur'anic Reading Comprehension Dataset): Combined train and development splits (169 queries) were used. The dataset provides exhaustive answers and reflects contemporary user information needs.
  • Quranic Texts: Verified translations of the Holy Quran in Arabic, English (Sahih International), Russian (Elmir Kuliev), and Urdu (Ahmed Raza Khan) were used as retrieval collections.

Key Insights and Ideas

  • Domain Adaptation is Crucial: Even a short round of pre-training on domain-specific data (like the Islamic corpus) significantly boosts retrieval performance, overcoming the typical domain shift problem.
  • Lightweight Models are Viable: Reducing the size of LLMs through language reduction and efficient training makes them practical for real-world deployment, especially in resource-constrained environments.
  • Mixed Training Offers a Sweet Spot: The proposed "mixed training" approach, which strategically combines monolingual and cross-lingual training data, consistently yields strong results across various evaluation types, outperforming purely monolingual or cross-lingual methods.
  • Embedding Space Matters: Visualizations (t-SNE) and heat maps reveal that effective training strategies lead to more unified embedding spaces, where language becomes less of a barrier for retrieval. Models trained with domain adaptation and mixed approaches show better cross-lingual retrieval capabilities.
  • Cost-Efficiency is a Major Win: Deploying a single, lightweight, versatile model can reduce costs by approximately 70% compared to multiple larger models, opening up possibilities for CPU-only serving and serverless architectures.

Key Results

  • Performance: The "Biq-ENc" model (trained with a mixed approach) consistently demonstrated strong performance across monolingual, multilingual, and cross-lingual evaluations. It achieved the highest average performance in monolingual evaluation (0.426) and was a top performer in Arabic and Urdu.
  • Multilingual Performance: In multilingual evaluation, the "4lingual-train-all" approach achieved the best average score (0.357), while "Biq-ENc" also performed well (0.354 average).
  • Cross-lingual Performance: "Biq-ENc" was the top performer for the EN-AR pair (0.423) and showed strong results for other pairs.
  • Model Size Reduction: The XLM-R4 model was reduced to 481 MB from 1.1 GB, a significant decrease that aids deployment.
  • Latency Improvement: Real-user monitoring showed substantial latency drops across regions after deployment, with median latency decreasing by up to 47.4% in APAC.
  • Cost Reduction: Deploying a single lightweight model (400 MB) can reduce costs by about 70% compared to three larger models.

Limitations and Future Work

  • Language Scalability: While the experiment included low-resource languages like Urdu, results might vary with a significantly larger number of languages.
  • Parallel Corpus Dependency: The mixed and cross-lingual training setups rely on parallel corpora, which may not be available or may be noisy in all scenarios.
  • Resource-Constrained Devices: The performance and efficiency trade-offs for truly resource-constrained devices (e.g., mobile or edge environments) require further exploration.
  • Future Work: The authors suggest exploring more languages, investigating alternative training data sources, and conducting further evaluations on edge devices. They also emphasize the importance of responsible deployment, especially for religious texts, and recommend involving domain experts and community stakeholders.

Summary updated

BibTeX
@inproceedings{pavlova2025efficient,
  author = {Vera Pavlova and Mohammed Makhlouf},
  title = {{Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure