TL;DR
This paper introduces Nomic Embed v2, the first general-purpose Mixture of Experts (MoE) model for text embeddings. It achieves state-of-the-art performance on both English and multilingual retrieval benchmarks while using significantly fewer active parameters than dense models, addressing critical deployment challenges like latency and memory usage.
What Problem Does This Paper Address?
Transformer-based text embedding models have become incredibly powerful, but their performance gains often come from simply scaling up the number of parameters. This leads to massive models that are slow and memory-hungry, making them impractical for real-world applications, especially in retrieval-augmented generation (RAG) systems where efficiency is paramount. While Mixture of Experts (MoE) architectures have shown promise in improving efficiency for language models, they haven't been effectively applied to the specific task of text embeddings. This paper aims to bridge that gap by developing the first general-purpose MoE text embedding model.
Methods
The core innovation is the application of the Mixture of Experts (MoE) architecture to text embedding models. MoE models work by having multiple "expert" sub-networks and a "router" that directs each input token to a small subset of these experts. This means only a fraction of the model's total parameters are active for any given input, leading to significant efficiency gains.
The authors adapted XLM-Roberta, a strong multilingual transformer, by replacing some of its standard feed-forward layers with MoE blocks. They used Top-K routing for the experts and incorporated an auxiliary load balancing loss to prevent expert collapse (where some experts get overloaded while others are underutilized). For training, they employed a two-stage process: weakly-supervised contrastive pretraining on a massive dataset, followed by contrastive finetuning with hard negative mining. They also incorporated Matryoshka Representation Learning to allow for flexible embedding dimensions.
Datasets
The paper utilizes several large-scale datasets for training and evaluation:
- Weakly-supervised contrastive pretraining: A massive dataset of 1.6 billion pairs derived from sources like mC4 and Multilingual CC News, covering 110 languages.
- Contrastive finetuning: Datasets from BEIR, MIRACL, SQUAD, and Stackoverflow.
- Evaluation Benchmarks:
- BEIR: A collection of English-only retrieval datasets.
- MIRACL: A multilingual retrieval benchmark.
- GLUE: A suite of natural language understanding tasks.
- XTREME-R: A comprehensive multilingual natural language understanding benchmark.
Key Insights and Ideas
- MoE for Embeddings is Viable: The paper demonstrates that MoE architectures, previously more common in generative language models, can be successfully adapted for text embedding tasks, offering significant efficiency benefits.
- Efficiency Without Sacrificing Performance: Nomic Embed v2 achieves competitive or superior performance compared to dense models of similar or even larger sizes, proving that scaling can be done more efficiently.
- Multilingual MoE Challenges: While MoE shows promise for multilingual embeddings, the paper notes that data scale significantly impacts performance, and MoE models might underperform dense counterparts in limited training regimes.
- Selective Layer Upcycling: Ablation studies suggest that converting only a subset of layers to MoE (e.g., 6 out of 12) can provide a better balance between model capacity and training stability than converting all layers.
Key Results
- Monolingual Performance: Nomic Embed v2 (305M parameters) outperforms similarly sized dense models and matches the performance of models twice its size on BEIR and MIRACL benchmarks.
- Multilingual Performance: Nomic Embed v2 achieves competitive performance on MIRACL, outperforming all multilingual models on BEIR except Arctic Embed v2 Large.
- GLUE Benchmark: The MoE variant of XLM-Roberta (mNomic-BERT) achieved comparable performance to XLM-R-Base across GLUE tasks, requiring significantly fewer pretraining steps than mGTE-Base.
- XTREME-R Benchmark: mNomic-BERT achieved an average score comparable to XLM-R-Base.
- Efficiency Gains: The MoE architecture allows for a model with 475M total parameters to have only 305M active parameters during inference, leading to reduced computational costs.
Limitations and Future Work
- Multilingual Data Scale: The paper acknowledges that multilingual MoE performance is highly dependent on data scale, and further research is needed to understand optimal training regimes for multilingual MoE embeddings.
- Optimal MoE Configuration: The authors suggest exploring the optimal scaling of expert count and active parameters, as well as alternative routing mechanisms.
- Distillation: Investigating techniques to distill MoE models back into dense architectures could make these efficiency gains more widely deployable.
- Scaling Relationships: Further research is needed to understand the fundamental scaling relationships between dataset size, model parameters, and embedding dimension for MoE models.
Summary updated