Evaluating Embedding APIs for Information Retrieval

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, Jimmy Lin

Abstract

The ever-increasing size of language models curtails their widespread availability to the community, thereby galvanizing many companies into offering access to large language models through APIs. One particular type, suitable for dense retrieval, is a semantic embedding service that builds vector representations of input text. With a growing number of publicly available APIs, our goal in this paper is to analyze existing offerings in realistic retrieval scenarios, to assist practitioners and researchers in finding suitable services according to their needs. Specifically, we investigate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval. For this purpose, we evaluate these services on two standard benchmarks, BEIR and MIRACL. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English, in contrast to the standard practice of employing them as first-stage retrievers. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost. We hope our work lays the groundwork for evaluating semantic embedding APIs that are critical in search and more broadly, for information access.

Introduction

As Large Language Models (LLMs) grow too massive for most organizations to host, proprietary embedding APIs from companies like OpenAI and Cohere have become the primary solution for semantic search. Modern Information Retrieval (IR) has transitioned from keyword matching to capturing semantic meaning through dense retrieval. However, the state-of-the-art models required for this are computationally intensive. Consequently, providers offer semantic embedding APIs that convert text into high-dimensional vectors for a fee.

Practitioners face two primary challenges: the "Black Box Problem," where training methodologies and generalization capabilities to niche domains or diverse languages remain opaque, and the "Cost-Benefit Problem," as indexing millions of documents via API can be prohibitively expensive. This study benchmarks these services to determine their performance relative to open-source alternatives and identifies the most cost-effective deployment strategies.

Methodology

The researchers evaluated three major API providers: OpenAI (text-embedding-ada-002), Cohere (small, large, and multilingual), and Aleph Alpha (luminous-base). Three primary architectural patterns were tested:

  1. Full-ranking: Standard dense retrieval where the entire document collection and queries are embedded.
  2. Top-100 Re-ranking: A hybrid approach using traditional BM25 search to find the top 100 candidates, which are then re-ordered by the API.
  3. Hybrid Retrieval: Combining lexical scores (BM25) and semantic scores (API) for a comprehensive ranking.

Evaluation was conducted on two standard benchmarks: BEIR, a heterogeneous meta-benchmark for English domain generalization across 18 datasets, and MIRACL, a large-scale multilingual benchmark covering 18 diverse languages.

Key Results and Discussion

The study reveals that re-ranking is often the most effective strategy. For English tasks, re-ranking the top 100 BM25 results with an API is significantly more cost-effective and frequently more accurate than full-ranking. However, embedding APIs still struggle with datasets requiring exact keyword matching, such as medical journals (BioASQ) or tweets (Signal1M), where BM25 remains competitive.

Table 1: Performance comparison on the BEIR benchmark (English) across various domains and retrieval strategies.

As shown in Table 1, OpenAI’s ada-002 achieved the highest average nDCG@10 of 0.500 on the BEIR benchmark, notably outperforming the open-source TASB model.

For multilingual retrieval, a hybrid approach is essential. While APIs are particularly effective for low-resource languages where training data is scarce, combining them with BM25 yields the best results overall.

Table 2: Performance comparison on the MIRACL benchmark (Multilingual) across 18 languages.

On the MIRACL benchmark (Table 2), the Cohere + BM25 hybrid model achieved the highest average score of 0.579 nDCG@10. The study also noted a significant cost disparity; for instance, re-ranking the BEIR benchmark cost approximately $170 via OpenAI compared to $2,500 via Cohere at the time of evaluation.

Conclusion and Future Work

Proprietary embedding APIs offer powerful retrieval capabilities, especially when used as re-rankers or in hybrid configurations. However, their proprietary nature presents challenges: models can be updated without notice, and a lack of transparency regarding training data raises concerns about data leakage and social bias. Future research should focus on evaluating the fairness and robustness of these services in sensitive real-world applications.

Summary updated

Abstract

The ever-increasing size of language models curtails their widespread availability to the community, thereby galvanizing many companies into offering access to large language models through APIs. One particular type, suitable for dense retrieval, is a semantic embedding service that builds vector representations of input text. With a growing number of publicly available APIs, our goal in this paper is to analyze existing offerings in realistic retrieval scenarios, to assist practitioners and researchers in finding suitable services according to their needs. Specifically, we investigate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval. For this purpose, we evaluate these services on two standard benchmarks, BEIR and MIRACL. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English, in contrast to the standard practice of employing them as first-stage retrievers. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost. We hope our work lays the groundwork for evaluating semantic embedding APIs that are critical in search and more broadly, for information access.
BibTeX
@inproceedings{kamalloo2023evaluating,
  author = {Ehsan Kamalloo and Xinyu Zhang and Odunayo Ogundepo and Nandan Thakur and David Alfonso-Hermelo and Mehdi Rezagholizadeh and Jimmy Lin},
  title = {{Evaluating Embedding APIs for Information Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.50},
}
Figure