Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

Luke Merrick, Danmei Xu, Gaurav Nuti, Daniel CamposarXiv 2024 · DOI · arXiv

Introduction

In the landscape of Retrieval-Augmented Generation (RAG), text embeddings serve as the critical link between user queries and relevant document retrieval. Historically, developers have faced a trade-off: utilizing high-performing but costly proprietary APIs or deploying massive open-source models (often exceeding 7 billion parameters) that are computationally expensive and slow for production use. Snowflake's Arctic-Embed addresses this "missing middle" by providing a suite of five open-source models ranging from 22M to 334M parameters. These models demonstrate that superior retrieval performance can be achieved through meticulous data curation and training strategies rather than raw parameter scaling.

Methodology and Data Strategy

The Arctic-Embed models utilize a BERT-like encoder architecture and follow a two-stage training pipeline designed for maximum efficiency and accuracy.

Large-Scale Contrastive Pre-training

The first stage involves training on a curated dataset of approximately 300 million pairs using InfoNCE loss. A key innovation here is Source Stratification: instead of a random mix, each minibatch is filled from a single data source, which significantly improves training stability and final model quality. The pre-training data is a diverse mix of web crawls, academic titles, and query-answer pairs.

Composition of the pre-training dataset, highlighting the diversity of sources used for stratification.

Quality-Focused Fine-tuning

The second stage refines the models on a high-quality dataset of ~1 million samples. This stage introduces "grounded synthetic data," where LLMs generate queries based on actual documents while being shown "hard negatives" (similar but irrelevant documents). This prevents the generation of overly simplistic queries and forces the model to learn finer semantic distinctions.

Key Results and Performance

Arctic-Embed models consistently punch above their weight class, setting new benchmarks for their respective sizes on the Massive Text Embedding Benchmark (MTEB) Retrieval leaderboard.

Arctic-Embed models (blue) achieve superior retrieval performance compared to other model families across various parameter scales on the MTEB benchmark.
  • Efficiency: The 334M parameter arctic-embed-l outperforms significantly larger proprietary models like OpenAI’s text-embed-3-large and Cohere’s embed-v3 in retrieval tasks.
  • Throughput: The medium variant (arctic-embed-m) is highly optimized for production, achieving a throughput of over 5,000 documents per second on 8x NVIDIA H100 GPUs.
  • SOTA Status: At release, every variant in the suite achieved state-of-the-art (SOTA) performance for its parameter class.

Insights and Discussion

The success of Arctic-Embed underscores several critical findings in embedding research:

  1. Data Organization Over Scale: How data is sampled and how negatives are mined is more impactful than the sheer volume of training pairs.
  2. The Hard Negative Balance: While hard negatives are essential, those that are "too hard" (effectively false negatives) can degrade performance. Arctic-Embed uses tunable thresholds to mitigate this.
  3. Pooling Strategies: Ablation studies indicated that [CLS] token pooling outperformed standard mean pooling for Semantic Text Similarity (STS) tasks in this architecture.

Future Directions

While Arctic-Embed excels in standard retrieval, there remains room for growth in long-context optimization (8k+ tokens), where it currently trails specialized long-context models. Future iterations will explore more robust curriculum learning—progressively introducing harder negatives during training—and model compression techniques like quantization and binarization to further enhance edge deployment capabilities.

Summary updated

BibTeX
@misc{merrick2024arctic,
  author = {Luke Merrick and Danmei Xu and Gaurav Nuti and Daniel Campos},
  title = {{Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models}},
  year = {2024},
  eprint = {2405.05374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.05374},
}
Figure