TL;DR
This paper introduces Compass-Embedding v4, a new multilingual embedding model designed for e-commerce, especially in low-resource languages. It tackles challenges like noisy training data and the need for fast, efficient inference, achieving state-of-the-art results in Southeast Asian languages and e-commerce tasks.
What Problem Does This Paper Address?
In the booming world of e-commerce, especially in emerging markets, there's a critical shortage of high-quality semantic representations for less common languages. This "low-resource language bottleneck" hinders essential applications like product search, recommendations, and retrieval systems. Existing embedding models often perform poorly in these scenarios due to:
- Data Scarcity and Noise: Low-resource languages have limited, often noisy, and unevenly distributed training data.
- Domain Specificity: E-commerce data is distinct from general web text, being semi-structured and entity-centric, which general-purpose models struggle with.
- Training Challenges: Large-batch contrastive learning, a common training method, can introduce "false negatives" – semantically similar items incorrectly treated as dissimilar – degrading performance.
- Production Constraints: Real-world deployment demands high-speed, low-latency inference without sacrificing accuracy.
Compass-Embedding v4 aims to bridge this gap by creating robust, efficient, and domain-specific multilingual embeddings.
Methods
The authors introduce three key innovations to address the challenges:
- Class-Aware Masking (CAM): This is a novel modification to the standard InfoNCE contrastive learning objective. During training, CAM intelligently masks out "false negatives" that belong to the same latent class (e.g., same product category). This prevents the model from being penalized for correctly identifying similar items, leading to better semantic discrimination without increasing computational cost. They also use Spherical Model Merging (SLERP) to combine different model checkpoints, helping to retain performance on general tasks while gaining domain-specific expertise, thus mitigating catastrophic forgetting.
- Diversified Data Construction: To combat data scarcity, they built a rich training corpus by combining:
- Public multilingual corpora.
- Context-grounded synthetic data generation using LLMs.
- Cross-lingual translation of existing datasets.
-
Proprietary e-commerce data. This approach prioritizes data quality, alignment, and contrastive utility over sheer volume, especially for low-resource languages.
-
Production-Ready Inference: To meet strict deployment requirements, they optimized the model for high-throughput, low-latency inference. This involves using techniques like vLLM for serving and FP8 quantization to reduce computational load and memory footprint, making the model cost-efficient.
Datasets
The paper utilizes a combination of datasets for training and evaluation:
- Training Data: A massive corpus of 3.57 million samples, comprising:
- Public corpora.
- Synthetic data generated by LLMs (detailed in Section 4.2.2).
- Proprietary e-commerce data (detailed in Section 4.3).
- Translated datasets, including SQuAD translated into SEA languages.
- Evaluation Benchmarks:
- SEA Languages and Portuguese: A comprehensive multilingual benchmark with 19 tasks from the Massive Text Embedding Benchmark (MTEB) ecosystem, covering Retrieval, Classification, Bitext Mining, and Natural Language Inference (NLI).
- E-commerce Tasks: A proprietary dataset with six specific tasks designed to capture e-commerce domain concepts (e.g., Query-Item Retrieval, Item-to-Item Recommendation).
- English Benchmarks: The English portion of the MTEB collection for evaluating generalist capabilities.
Key Insights and Ideas
- False Negatives are a Systematic Bias: The paper argues that false negatives in large-batch contrastive learning aren't just random noise but a systematic issue arising from heterogeneous data sources. CAM directly addresses this by leveraging class metadata.
- Synthetic Data is Crucial for Low-Resource Languages: The success in SEA languages heavily relies on a carefully constructed synthetic data pipeline that goes beyond simple generation to ensure contextual grounding and realism.
- Balancing Domain Specialization and Generalization is Key: The SLERP model merging technique is vital for preventing performance degradation on general tasks (like English) while achieving strong results in specialized domains (like e-commerce).
- Inference Efficiency is as Important as Accuracy: The paper demonstrates that significant gains in throughput and cost reduction can be achieved through optimizations like FP8 quantization and prefix caching, making advanced models practical for large-scale deployment.
Key Results
- State-of-the-Art Performance: Compass-Embedding v4 achieves state-of-the-art results on major SEA languages, outperforming strong open-source models like Qwen3-Embedding-4B and proprietary models like OpenAI's Text-Embedding-3 series by significant margins (e.g., 10.5% and 26.6% lead in SEA languages).
- E-commerce Dominance: It shows a 16.2% generation-over-generation improvement on their proprietary e-commerce tasks compared to v3, establishing a new baseline and outperforming Qwen3-Embedding-4B by 9.7%.
- Inference Efficiency: Achieves an over 8x increase in inference throughput and an 88% reduction in cost compared to Compass-Embedding v3, costing only 15% of OpenAI's text-embedding-3-small under identical SLO constraints.
- Minimal Accuracy Trade-off with Quantization: FP8 quantization resulted in minimal accuracy drops (average of 0.13% for per-tensor FP8), enabling significant speedups.
Limitations and Future Work
- Limitations: While the paper highlights significant improvements, it acknowledges that performance on high-resource languages is maintained but not necessarily pushed to the absolute cutting edge compared to models solely focused on them. The evaluation of FP8 quantization showed minimal accuracy degradation, but it's still a trade-off to consider.
- Future Work: The authors suggest several promising directions:
- More fine-grained product representations.
- Multimodal embeddings integrating text and visual signals.
- Deeper and more robust multilingual alignment.
- Embedding representations enhanced with lightweight reasoning capabilities.
- Further research into next-generation multilingual retrieval, recommendation, and RAG systems.
Summary updated