Introduction
For the vast majority of the world's languages, the lack of digital text corpora prevents the training of standard large language models (LLMs). However, many of these low-resource languages are documented in descriptive grammars—comprehensive books detailing linguistic rules. This paper introduces a framework for "Retrieval-Augmented Classification" (RAC), aiming to move from general text embeddings (MTEB) toward the goal of "Machine Translation from One Book" (MTOB). By using RAG to extract typological features from dense, technical grammars, the authors demonstrate how LLMs can support languages lacking traditional training data.
The Challenge of Descriptive Grammars
Descriptive grammars are written for human linguists, featuring specialized terminology and information scattered across hundreds of pages. Current LLMs struggle to process these books zero-shot. This research addresses the gap in systematically extracting structured linguistic data (typological features) from these complex texts to power NLP tools for underserved communities.
Methodology: Retrieval-Augmented Classification
The proposed Advanced RAG pipeline is specifically tuned for linguistic classification. Instead of open-ended generation, the pipeline performs classification by selecting a feature value (e.g., "Subject-Verb-Object") from a closed set based on retrieved context.
The pipeline includes:
- Chunking: Grammars are split into paragraph-sized units.
- Retrieval & Reranking: A hybrid approach uses BM25 as a baseline, followed by state-of-the-art embedding models (e.g.,
SFR-Embedding-Mistral) to rerank the top 50 paragraphs down to the 20 most relevant. - Prompting Strategies: The study compares a baseline prompt against Chain-of-Thought (CoT) prompts that include detailed linguistic guidelines and examples from databases like WALS and Grambank.
- LLM Classifier: GPT-4o determines the final typological value based on the retrieved context.
Datasets and Benchmarks
Two new benchmarks were introduced:
- Benchmark for Rerankers: 700 paragraphs from 14 grammars, manually annotated for relevance to word order (WALS 81A).
- Benchmark for RAG: A collection of 148 descriptive grammars stratified by macroarea to ensure geographic and familial diversity. It evaluates four complex features: Word Order, Standard Negation, Polar Questions, and Number of Cases.
Analysis of Retrieval Performance
A critical finding is that performance on general benchmarks like the Massive Text Embedding Benchmark (MTEB) does not correlate with success in the linguistic domain.
As shown in the table above, models ranking high on MTEB often underperform on descriptive grammars. In fact, a strong negative correlation was observed. This suggests that the dense, technical nature of linguistic description requires specialized retrieval capabilities that general-purpose models lack. Furthermore, the classic BM25 algorithm remains highly competitive, ranking 6th out of 8 models.
Key Results
The RAG configurations significantly outperformed the baseline GPT-4o model without grammar context.
On Word Order (WALS 81A), the macro F1 score improved from 0.28 (baseline) to 0.71 (BM25). The SFR-Embedding-Mistral model emerged as the top reranker when combined with specific instructions and Wikipedia-based queries. However, quantitative features like "Number of Cases" (WALS 49A) remained challenging, as relevant information is often distributed across multiple chapters.
Discussion and Future Work
The study highlights that retrieval is the primary bottleneck. An ablation study using human-selected "perfect" paragraphs showed a massive performance jump, indicating that the LLM's reasoning is sufficient if the correct context is provided.
While Chain-of-Thought (CoT) prompting is often beneficial, it yielded inconsistent results here, sometimes confusing the model with overly specific instructions. Future work will focus on optimizing the retrieval process, expanding the language coverage beyond the current 148 languages, and applying these extracted features to improve zero-shot machine translation for extremely low-resource settings.
Summary updated