TL;DR
This paper introduces TR-MTEB, the first comprehensive benchmark and model suite for evaluating Turkish sentence embeddings. By creating a large-scale, task-diverse benchmark and training Turkish-specific embedding models, they aim to address the underrepresentation of low-resource languages like Turkish in NLP research and promote more equitable benchmarking.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen tremendous progress, largely driven by advancements in text embeddings. These embeddings allow computers to understand and represent text in a way that captures semantic meaning. However, most research and development in this area have focused on high-resource languages like English, leaving low-resource languages, such as Turkish, significantly underrepresented. This lack of dedicated benchmarks and models for Turkish hinders the development of robust Turkish NLP systems and makes it difficult to fairly evaluate the performance of multilingual models on Turkish text. Existing benchmarks, while valuable, often lack the task diversity and language-specific nuances needed for a thorough evaluation of Turkish sentence embeddings.
Methods
The authors tackle this problem in two main ways:
- TR-MTEB Benchmark Creation: They assembled a comprehensive benchmark for Turkish sentence embeddings, named TR-MTEB. This involved:
- Curating Existing Datasets: They gathered high-quality, publicly available Turkish datasets across six core NLP tasks: classification, clustering, retrieval, bitext mining, pair classification, and semantic textual similarity.
- Translating Key Datasets: To expand coverage, especially for retrieval tasks where Turkish datasets are scarce, they translated several established English datasets (from the BEIR benchmark) into Turkish. This translation process involved using LLMs and a rigorous LLM-as-a-Judge evaluation pipeline to ensure semantic fidelity and fluency.
- Training Turkish-Specific Embedding Models: To complement the benchmark, they trained two Turkish-specific sentence embedding models.
- Corpus Creation: They curated a large corpus of 34.2 million weakly supervised Turkish sentence pairs from various public sources. This corpus was filtered using a fine-tuned embedding model to ensure quality.
- Model Training: They trained models initialized from the BERTurk base model. This involved:
- Contrastive Pretraining: Using Cached Multiple Negatives Ranking Loss (Cached-MNRL) on the large weakly supervised corpus for efficient contrastive learning.
- Supervised Fine-tuning: Further fine-tuning the pretrained models on task-specific subsets of the TR-MTEB benchmark to adapt them to specific task objectives.
Datasets
The paper introduces TR-MTEB, a new benchmark comprising 26 datasets across six tasks. These datasets include:
- Classification: 8 datasets (e.g., THY Sentiment, Turkish News Category, Turkish Offensive Language).
- Clustering: 2 datasets (e.g., Turkish Abstract Corpus, Turkish Column Writing).
- Pair Classification: 3 datasets (MNLI TR, SNLI TR, XNLI Turkish Split).
- Bitext Mining: 1 dataset (WMT16 EN-TR).
- Semantic Textual Similarity (STS): 1 dataset (STSb TR).
- Retrieval: 11 datasets (e.g., SQUAD TR, TQUAD, XQuAD, MSMARCO TR, NFCorpus TR).
For training their models, they created a corpus of 34.2 million weakly supervised Turkish sentence pairs by combining data from over 50 publicly available Turkish-language resources.
Key Insights and Ideas
- Low-Resource Languages Need Dedicated Benchmarks: The success of language-specific benchmarks like C-MTEB and PL-MTEB highlights the critical need for tailored resources for languages like Turkish.
- Weak Supervision and Contrastive Learning are Powerful: Training on a large, weakly supervised corpus using contrastive learning can yield competitive results even with limited resources, demonstrating the effectiveness of modern self-supervised techniques.
- Monolingual Models Can Be Competitive: Their Turkish-specific models, despite being smaller and trained on less data than large multilingual models, perform competitively on many monolingual tasks, showcasing the value of language-specific optimization.
- Cross-Lingual Tasks Remain a Challenge for Monolingual Models: The significant performance gap in bitext mining for their monolingual models underscores the inherent limitations of not having cross-lingual pretraining for tasks requiring alignment between languages.
- Fine-tuning Boosts Performance: Supervised fine-tuning on task-specific data significantly improves performance, especially in tasks like bitext mining and pair classification, demonstrating the benefit of adapting general models to specific downstream objectives.
Key Results
- The authors' fine-tuned Turkish embedding model achieved a Mean (Task) score of 63.42 and Mean (Type) score of 70.25 on the TR-MTEB benchmark.
- Their models significantly outperform the base BERTurk model and perform competitively on monolingual tasks like classification and STS.
- The fine-tuned model showed substantial gains over the contrastive-only version, with improvements of +49.2 points in Bitext Mining, +13.4 points in Pair Classification, and +3.5 points in STS.
- While their Turkish-specific model (110M parameters) narrows the gap with larger multilingual models (up to 560M parameters), the latter still lead in overall performance, particularly in cross-lingual tasks.
Limitations and Future Work
- Reliance on Machine Translation: A significant portion of TR-MTEB consists of machine-translated datasets. While efforts were made to ensure quality, these translations might still introduce subtle artifacts or differ from native Turkish in syntactic structure or idiomatic expressions.
- Weak Supervision Noise: The 34.2 million sentence pair corpus, while large, is based on weak supervision and may contain noise or spurious patterns, potentially limiting the upper bound of model performance. The corpus size is also relatively small compared to hundreds of millions or billions of pairs used in high-resource language studies.
- Model Capacity: The Turkish embedding models were trained on a relatively lightweight BERTurk encoder, limiting their parameter capacity compared to state-of-the-art multilingual models.
- Task Coverage: TR-MTEB currently does not include tasks like summarization, re-ranking, or question generation.
Future work includes scaling up the training data, incorporating multilingual contrastive objectives for better cross-lingual generalization, and extending TR-MTEB with more tasks and native Turkish datasets.
Summary updated