From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

Albert Kornilov, Tatiana ShavrinaarXiv 2024 · DOI · arXiv

Introduction

For the vast majority of the world's languages, the lack of digital text corpora prevents the training of standard large language models (LLMs). However, many of these low-resource languages are documented in descriptive grammars—comprehensive books detailing linguistic rules. This paper introduces a framework for "Retrieval-Augmented Classification" (RAC), aiming to move from general text embeddings (MTEB) toward the goal of "Machine Translation from One Book" (MTOB). By using RAG to extract typological features from dense, technical grammars, the authors demonstrate how LLMs can support languages lacking traditional training data.

The Challenge of Descriptive Grammars

Descriptive grammars are written for human linguists, featuring specialized terminology and information scattered across hundreds of pages. Current LLMs struggle to process these books zero-shot. This research addresses the gap in systematically extracting structured linguistic data (typological features) from these complex texts to power NLP tools for underserved communities.

Methodology: Retrieval-Augmented Classification

The proposed Advanced RAG pipeline is specifically tuned for linguistic classification. Instead of open-ended generation, the pipeline performs classification by selecting a feature value (e.g., "Subject-Verb-Object") from a closed set based on retrieved context.

The pipeline includes:

  • Chunking: Grammars are split into paragraph-sized units.
  • Retrieval & Reranking: A hybrid approach uses BM25 as a baseline, followed by state-of-the-art embedding models (e.g., SFR-Embedding-Mistral) to rerank the top 50 paragraphs down to the 20 most relevant.
  • Prompting Strategies: The study compares a baseline prompt against Chain-of-Thought (CoT) prompts that include detailed linguistic guidelines and examples from databases like WALS and Grambank.
  • LLM Classifier: GPT-4o determines the final typological value based on the retrieved context.

Datasets and Benchmarks

Two new benchmarks were introduced:

  1. Benchmark for Rerankers: 700 paragraphs from 14 grammars, manually annotated for relevance to word order (WALS 81A).
  2. Benchmark for RAG: A collection of 148 descriptive grammars stratified by macroarea to ensure geographic and familial diversity. It evaluates four complex features: Word Order, Standard Negation, Polar Questions, and Number of Cases.

Analysis of Retrieval Performance

A critical finding is that performance on general benchmarks like the Massive Text Embedding Benchmark (MTEB) does not correlate with success in the linguistic domain.

Comparison of MTEB and Grammar Benchmark rankings across various models.

As shown in the table above, models ranking high on MTEB often underperform on descriptive grammars. In fact, a strong negative correlation was observed. This suggests that the dense, technical nature of linguistic description requires specialized retrieval capabilities that general-purpose models lack. Furthermore, the classic BM25 algorithm remains highly competitive, ranking 6th out of 8 models.

Key Results

The RAG configurations significantly outperformed the baseline GPT-4o model without grammar context.

F1 mean scores for various linguistic features across different model configurations.

On Word Order (WALS 81A), the macro F1 score improved from 0.28 (baseline) to 0.71 (BM25). The SFR-Embedding-Mistral model emerged as the top reranker when combined with specific instructions and Wikipedia-based queries. However, quantitative features like "Number of Cases" (WALS 49A) remained challenging, as relevant information is often distributed across multiple chapters.

Discussion and Future Work

The study highlights that retrieval is the primary bottleneck. An ablation study using human-selected "perfect" paragraphs showed a massive performance jump, indicating that the LLM's reasoning is sufficient if the correct context is provided.

While Chain-of-Thought (CoT) prompting is often beneficial, it yielded inconsistent results here, sometimes confusing the model with overly specific instructions. Future work will focus on optimizing the retrieval process, expanding the language coverage beyond the current 148 languages, and applying these extracted features to improve zero-shot machine translation for extremely low-resource settings.

Summary updated

Abstract

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at \url{https://github.com/al-the-eigenvalue/RAG-on-grammars}.
BibTeX
@misc{kornilov2024from,
  author = {Albert Kornilov and Tatiana Shavrina},
  title = {{From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars}},
  year = {2024},
  eprint = {2411.15577},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15577},
}
Figure