Evaluating Embedding APIs for Information Retrieval

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, Jimmy Lin

Abstract

Commercial embedding APIs have become a popular solution for implementing dense retrieval in production environments. However, their "black-box" nature and the lack of independent benchmarking pose challenges for practitioners. This paper evaluates major embedding services from OpenAI, Cohere, and Aleph Alpha across diverse domains and languages. By benchmarking these models on the BEIR and MIRACL datasets, we demonstrate that a re-ranking strategy using these APIs significantly enhances traditional keyword search. Furthermore, we find that while OpenAI's models excel in English-language tasks, a hybrid approach combining BM25 with multilingual embeddings like Cohere's is essential for optimal performance in non-English and low-resource settings.

Introduction

The advent of Large Language Models (LLMs) has significantly advanced the field of Information Retrieval (IR), particularly through dense retrieval techniques that capture semantic meaning. Despite these gains, the computational cost of hosting state-of-the-art open-source models remains a barrier for many organizations. Proprietary "Embedding-as-a-Service" (EaaS) APIs offer a scalable alternative, yet they operate as black boxes with unknown training data and architectures. This study provides a rigorous evaluation of these commercial APIs to determine their generalizability across various domains and their effectiveness in multilingual contexts.

Methodology

The researchers evaluated three prominent EaaS providers: OpenAI (text-embedding-ada-002), Cohere (large, small, and multilingual-22-12), and Aleph Alpha (luminous-base). To maintain cost-efficiency, the study focused on a re-ranking paradigm rather than full-collection ranking. In this setup, the BM25 algorithm is first used to retrieve the top 100 candidate documents, which are then re-ordered based on semantic similarity scores provided by the APIs. For multilingual tasks, a hybrid approach was employed, integrating scores from both BM25 and the embedding APIs to leverage both lexical and semantic signals.

Benchmarks

Evaluation was conducted on two primary benchmarks:

  1. BEIR: A standard for zero-shot English IR evaluation, comprising 18 datasets across 9 domains (e.g., Bio-Medical, Finance, News).
  2. MIRACL: A large-scale multilingual benchmark covering 18 diverse languages, categorized by resource availability (high, medium, and low).

Results and Analysis

The evaluation revealed several key insights into the performance and cost-effectiveness of commercial embedding APIs.

English Retrieval Performance

OpenAI's ada2 model emerged as the overall leader on the BEIR benchmark, achieving an average nDCG@10 of 0.500. This significantly outperformed the traditional BM25 baseline (0.424) and the specialized open-source TASB model (0.414). However, the results also highlighted a persistent "lexical weakness" in dense models; on datasets requiring precise keyword matching (e.g., BioASQ), BM25 often remained superior, suggesting that semantic embeddings have not yet fully replaced keyword-heavy retrieval.

Table 4: Zero-shot evaluation on BEIR (nDCG@10). OpenAI's text-embedding-ada-002 shows strong performance across most English domains.

Multilingual Retrieval Performance

In the multilingual setting, the hybrid approach proved most effective. A combination of Cohere's multilingual embeddings and BM25 achieved an average nDCG@10 of 0.579, substantially higher than the best standalone dense retriever, mDPR (0.420). Notably, while well-tuned open-source models can compete in high-resource languages, proprietary APIs demonstrate a clear advantage in low-resource languages where training data is limited.

Table 5: Multilingual evaluation on MIRACL (nDCG@10). The hybrid Cohere+BM25 approach consistently outperforms standalone models across diverse languages.

Cost Considerations

Economic factors are a critical component of EaaS adoption. During the study period (February 2023), significant price disparities were observed. Re-ranking the BEIR benchmark cost approximately $170 via OpenAI, compared to roughly $2,500 via Cohere based on their then-current per-call pricing model, highlighting the importance of considering pricing alongside performance.

Discussion and Limitations

While commercial APIs offer high performance and ease of use, they represent a "moving target." Providers frequently update models and pricing, which can impact long-term reliability and cost-effectiveness. Additionally, this study did not address production-level latency or potential societal biases inherent in these black-box embeddings. Future research should focus on auditing these models for fairness and establishing benchmarks for real-world deployment latency.

Conclusion

Using commercial embedding APIs for re-ranking is a highly effective and practical strategy for modern IR systems. For English-centric applications, OpenAI's offerings provide a strong balance of performance and cost. For multilingual applications, a hybrid retrieval strategy incorporating multilingual embeddings is essential for achieving state-of-the-art results across a broad spectrum of languages.

Summary updated

Abstract

The ever-increasing size of language models curtails their widespread availability to the community, thereby galvanizing many companies into offering access to large language models through APIs. One particular type, suitable for dense retrieval, is a semantic embedding service that builds vector representations of input text. With a growing number of publicly available APIs, our goal in this paper is to analyze existing offerings in realistic retrieval scenarios, to assist practitioners and researchers in finding suitable services according to their needs. Specifically, we investigate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval. For this purpose, we evaluate these services on two standard benchmarks, BEIR and MIRACL. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English, in contrast to the standard practice of employing them as first-stage retrievers. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost. We hope our work lays the groundwork for evaluating semantic embedding APIs that are critical in search and more broadly, for information access.
BibTeX
@inproceedings{kamalloo2023evaluating,
  author = {Ehsan Kamalloo and Xinyu Zhang and Odunayo Ogundepo and Nandan Thakur and David Alfonso-Hermelo and Mehdi Rezagholizadeh and Jimmy Lin},
  title = {{Evaluating Embedding APIs for Information Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.50},
}
Figure