Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

Isaac Chung, Imene Kerboua, Márton Kardos, Roman Solomatin, Kenneth C. Enevoldsen

Introduction

The Massive Text Embedding Benchmark (MTEB) has established itself as the industry standard for evaluating how AI models represent text. However, as the ecosystem expanded to cover over 250 languages and specialized domains like chemistry and law, it faced significant engineering hurdles. This paper details the "under-the-hood" infrastructure required to maintain reproducibility and extensibility in such a massive ecosystem, introducing rigorous versioning, automated validation, and a novel "zero-shot score" to detect data contamination.

Methods

To address the challenges of "bit rot" and inconsistent reporting, the authors transitioned MTEB from a static collection of scripts to a robust, modular software framework. This architecture separates the system into distinct components: standardized model interfaces, model-agnostic task definitions, dataset handlers, and result processors.

The modular architecture of the MTEB framework, separating models, benchmarks, and results.

Key technical choices include:

  • Multi-level Versioning: Ensuring "apples-to-apples" comparisons by versioning code, specific dataset revisions (via Hugging Face), evaluation protocols, and model checkpoints.
  • Automated CI/CD: Every contribution undergoes a rigorous pipeline using Pydantic for metadata validation and mock tasks with synthetic models to test evaluation logic without high compute costs.
  • Reproducible Environments: The framework enforces consistent seeds for stochastic tasks and integrates tools like codecarbon to track the environmental impact of evaluation runs.

The MTEB Ecosystem

MTEB serves as a "meta-benchmark" aggregating hundreds of datasets across various paradigms, including classification, clustering, retrieval, and summarization. The infrastructure supports numerous specialized extensions:

  • C-MTEB (Chinese)
  • MMTEB (250+ languages)
  • ChemTEB (Domain-specific)
  • MIEB (Multi-modal)

A critical shift was moving from self-reported scores on model cards to a centralized, community-verified repository where results undergo peer review via Pull Requests.

Key Insights and Ideas

The "Zero-Shot" Score

A primary innovation is the transparency metric known as the "Zero-Shot" Score ($z = 1 - n_{train}/n_{total}$). This metric is based on the proportion of benchmark datasets a model was exposed to during its training phase. It allows users to distinguish between true "generalist" models and those that may have high performance due to memorizing test data (data contamination).

Mean performance vs. Zero-shot Score (z), illustrating how the transparency metric helps identify potential data contamination.

Handling Embedding Complexity

The authors found that reproducing reported scores is rarely straightforward. Many top-performing models require specific "prefixes" (e.g., adding "query: " before a search term), task-specific prompts, or custom LoRA adapters. The MTEB framework now explicitly supports these diverse requirements, including multi-stage encoding and custom inference parameters.

Key Results

The engineering practices detailed in this paper have enabled MTEB to remain the primary global reference for embedding model performance. The verified leaderboard has significantly increased trust in rankings by relying on a centralized, peer-reviewed repository. Furthermore, the framework's architectural flexibility now accommodates a wide landscape of models, including those with non-normalized embeddings and complex inference requirements.

Limitations and Future Work

While MTEB has successfully scaled, the authors identify several areas for future development:

  • Bias and Fairness: Developing standardized methods to measure social bias in embeddings.
  • Domain Gaps: Increasing coverage in underrepresented fields like health, arts, and culture.
  • Versioning Friction: Moving toward stricter backward compatibility and semantic versioning to reduce user frustration during refactors.
  • Documentation: Integrating all documentation into code docstrings to ensure it evolves in sync with the codebase.

Summary updated

Abstract

The Massive Text Embedding Benchmark (MTEB) has become a standard evaluation platform for text embedding models. While previous work has established the core benchmark methodology, this paper focuses on the engineering aspects that ensure MTEB's continued reproducibility and extensibility. We present our approach to maintaining robust continuous integration pipelines that validate dataset integrity, automate test execution, and assess benchmark results' generalizability. We detail the design choices that collectively enhance reproducibility and usability. Furthermore, we discuss our strategies for handling community contributions and extending the benchmark with new tasks and datasets. These engineering practices have been instrumental in scaling MTEB to become more comprehensive while maintaining quality and, ultimately, relevance to the field. Our experiences offer valuable insights for benchmark maintainers facing similar challenges in ensuring reproducibility and usability in machine learning evaluation frameworks. The MTEB repository is available at: \url{https://anonymous.4open.science/r/mteb-B78C}.
BibTeX
@inproceedings{chung2025maintaining,
  author = {Isaac Chung and Imene Kerboua and Márton Kardos and Roman Solomatin and Kenneth C. Enevoldsen},
  title = {{Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks}},
  booktitle = {CODEML @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=qcPJs0KRZW},
}
Figure