Abstract
Commercial embedding APIs have become a popular solution for implementing dense retrieval in production environments. However, their "black-box" nature and the lack of independent benchmarking pose challenges for practitioners. This paper evaluates major embedding services from OpenAI, Cohere, and Aleph Alpha across diverse domains and languages. By benchmarking these models on the BEIR and MIRACL datasets, we demonstrate that a re-ranking strategy using these APIs significantly enhances traditional keyword search. Furthermore, we find that while OpenAI's models excel in English-language tasks, a hybrid approach combining BM25 with multilingual embeddings like Cohere's is essential for optimal performance in non-English and low-resource settings.
Introduction
The advent of Large Language Models (LLMs) has significantly advanced the field of Information Retrieval (IR), particularly through dense retrieval techniques that capture semantic meaning. Despite these gains, the computational cost of hosting state-of-the-art open-source models remains a barrier for many organizations. Proprietary "Embedding-as-a-Service" (EaaS) APIs offer a scalable alternative, yet they operate as black boxes with unknown training data and architectures. This study provides a rigorous evaluation of these commercial APIs to determine their generalizability across various domains and their effectiveness in multilingual contexts.
Methodology
The researchers evaluated three prominent EaaS providers: OpenAI (text-embedding-ada-002), Cohere (large, small, and multilingual-22-12), and Aleph Alpha (luminous-base). To maintain cost-efficiency, the study focused on a re-ranking paradigm rather than full-collection ranking. In this setup, the BM25 algorithm is first used to retrieve the top 100 candidate documents, which are then re-ordered based on semantic similarity scores provided by the APIs. For multilingual tasks, a hybrid approach was employed, integrating scores from both BM25 and the embedding APIs to leverage both lexical and semantic signals.
Benchmarks
Evaluation was conducted on two primary benchmarks:
- BEIR: A standard for zero-shot English IR evaluation, comprising 18 datasets across 9 domains (e.g., Bio-Medical, Finance, News).
- MIRACL: A large-scale multilingual benchmark covering 18 diverse languages, categorized by resource availability (high, medium, and low).
Results and Analysis
The evaluation revealed several key insights into the performance and cost-effectiveness of commercial embedding APIs.
English Retrieval Performance
OpenAI's ada2 model emerged as the overall leader on the BEIR benchmark, achieving an average nDCG@10 of 0.500. This significantly outperformed the traditional BM25 baseline (0.424) and the specialized open-source TASB model (0.414). However, the results also highlighted a persistent "lexical weakness" in dense models; on datasets requiring precise keyword matching (e.g., BioASQ), BM25 often remained superior, suggesting that semantic embeddings have not yet fully replaced keyword-heavy retrieval.
Multilingual Retrieval Performance
In the multilingual setting, the hybrid approach proved most effective. A combination of Cohere's multilingual embeddings and BM25 achieved an average nDCG@10 of 0.579, substantially higher than the best standalone dense retriever, mDPR (0.420). Notably, while well-tuned open-source models can compete in high-resource languages, proprietary APIs demonstrate a clear advantage in low-resource languages where training data is limited.
Cost Considerations
Economic factors are a critical component of EaaS adoption. During the study period (February 2023), significant price disparities were observed. Re-ranking the BEIR benchmark cost approximately $170 via OpenAI, compared to roughly $2,500 via Cohere based on their then-current per-call pricing model, highlighting the importance of considering pricing alongside performance.
Discussion and Limitations
While commercial APIs offer high performance and ease of use, they represent a "moving target." Providers frequently update models and pricing, which can impact long-term reliability and cost-effectiveness. Additionally, this study did not address production-level latency or potential societal biases inherent in these black-box embeddings. Future research should focus on auditing these models for fairness and establishing benchmarks for real-world deployment latency.
Conclusion
Using commercial embedding APIs for re-ranking is a highly effective and practical strategy for modern IR systems. For English-centric applications, OpenAI's offerings provide a strong balance of performance and cost. For multilingual applications, a hybrid retrieval strategy incorporating multilingual embeddings is essential for achieving state-of-the-art results across a broad spectrum of languages.
Summary updated