FaMTEB: Massive Text Embedding Benchmark in Persian Language

Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, Arash AminiEMNLP 2025

TL;DR

This paper introduces FaMTEB, the first large-scale benchmark for evaluating Persian text embeddings, addressing the critical need for robust evaluation in low-resource languages. By building upon the MTEB framework and introducing 39 new Persian datasets across 7 tasks, FaMTEB provides a comprehensive resource for developing and assessing Persian NLP models, particularly for emerging applications like chatbots and RAG systems.

What Problem Does This Paper Address?

The field of text embeddings has seen rapid advancements, with models like BERT and its successors achieving remarkable performance on various Natural Language Processing (NLP) tasks. However, existing evaluation benchmarks, such as the Massive Text Embedding Benchmark (MTEB), are primarily focused on English. This leaves a significant gap in evaluating and advancing NLP capabilities for low-resource languages like Persian. The scarcity of comprehensive, task-specific datasets in Persian hinders the development and deployment of high-quality Persian NLP models, especially for critical applications like chatbots and Retrieval-Augmented Generation (RAG) systems, which rely heavily on effective text representations.

Methods

The core contribution of this work is the creation of FaMTEB, a Persian text embedding benchmark. This benchmark is built upon the existing MTEB framework, expanding its scope to include Persian language tasks. The authors introduce 39 new Persian datasets, which were generated through three primary approaches:

  • Web-based collection: Datasets were curated by scraping and labeling data from Persian websites.
  • Translation: Existing English datasets were translated into Persian using machine translation (primarily Google Translate) and then rigorously evaluated.
  • Large Language Model (LLM) generation: Synthetic datasets were created using LLMs like GPT-4o-mini, with carefully designed prompts to ensure diverse topics, tones, and linguistic phenomena.

The benchmark encompasses 7 distinct NLP tasks: classification, clustering, semantic textual similarity (STS), retrieval, reranking, pair classification, and a newly introduced task, summary retrieval. For evaluation, standard metrics from MTEB are used, with specific adaptations for the new summary retrieval task.

Datasets

FaMTEB comprises a total of 63 datasets. Of these, 24 were pre-existing in Persian, while 39 are newly introduced. These new datasets cover a wide range of NLP tasks and domains, including:

  • Synthetic Persian STS: A novel dataset for semantic textual similarity.
  • Synthetic Persian Chatbot: Datasets designed for various chatbot-related challenges, including conversation, RAG, and sentiment analysis.
  • Synthetic Persian Keywords/Tone/QA: Datasets for keyword extraction, tone analysis, and question answering.
  • Query to Query: A dataset for evaluating query similarity.
  • Clustering Datasets: Including BeytooteClustering and HamshahriClustring.
  • Translated Datasets: Notably, BEIR-Fa, a translated version of the BEIR benchmark, and mMARCO, a translated version of MS MARCO.

The paper also details the creation process for these datasets, emphasizing quality control through human and automated evaluation.

Key Insights and Ideas

  • The critical need for low-resource language benchmarks: The paper strongly highlights the gap in NLP evaluation for languages like Persian, emphasizing that English-centric benchmarks are insufficient for global NLP progress.
  • LLMs as powerful dataset generators: The successful creation of numerous high-quality synthetic Persian datasets using LLMs demonstrates their potential for addressing data scarcity in low-resource languages.
  • Summary Retrieval as a novel task: The introduction of "summary retrieval" as a new task within the benchmark is a significant contribution, addressing a specific need in evaluating models for dialogue summarization and retrieval.
  • Chatbot and RAG datasets are crucial: The inclusion of datasets specifically for chatbot and RAG systems reflects the growing importance of these applications and the need for specialized evaluation.
  • Quality of translated data is comparable: The evaluation of translated datasets shows that machine translation, when combined with rigorous quality checks, can produce reliable data for benchmark purposes.

Key Results

The paper evaluates 15 Persian and multilingual embedding models on the FaMTEB benchmark. While specific model performance details are presented in tables, a key finding is that the Jina-embeddings-v3 model achieves the highest overall accuracy on the FaMTEB benchmark, demonstrating strong performance across STS, retrieval, and summary retrieval tasks. The BGE-m3 model shows excellent performance in reranking and is identified as a tailored model for information retrieval. For classification, clustering, and pair classification tasks, Persian-specific models like sentence-transformer-parsbert, TookaBERT, and Tooka-SBERT achieve the highest accuracies. The evaluation of translated data using the BM25 metric indicates that Persian retrieval accuracy is comparable to English, suggesting the translated BEIR datasets meet quality standards.

Limitations and Future Work

The authors acknowledge several limitations:

  • Human-annotated datasets: The scarcity of human-annotated datasets in Persian remains a challenge, requiring extensive effort to create diverse and representative data.
  • Closed-source models: The high cost of accessing APIs for some closed-source models (e.g., text-embedding-3-large) has limited their inclusion in the current benchmark.
  • Machine translation quality: While Google Translate was used for cost-effectiveness, it may have limited the overall translation quality for some datasets.
  • Annotator bias: The paper authors themselves served as annotators, and while efforts were made to mitigate bias through collaborative guideline development, it remains a potential limitation.
  • Data leakage: The possibility of LLMs having been exposed to similar sentence structures or distributional patterns during training, even for synthetic data, is a concern.

Future work includes:

  • Reducing the size of selected BEIR datasets and exploring the use of more advanced LLMs (both closed and open-source) for translation to improve quality.
  • Expanding the benchmark by including results from a wider range of embedding models, particularly those based on LLMs, to enable more comprehensive evaluation across architectures and training paradigms.

Summary updated

BibTeX
@inproceedings{zinvandi2025famteb,
  author = {Erfan Zinvandi and Morteza Alikhani and Mehran Sarmadi and Zahra Pourbahman and Sepehr Arvin and Reza Kazemi and Arash Amini},
  title = {{FaMTEB: Massive Text Embedding Benchmark in Persian Language}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure