Introduction
UrBLiMP is the first large-scale diagnostic benchmark designed to evaluate the syntactic "common sense" of Large Language Models (LLMs) in Urdu. By using 5,696 minimal sentence pairs that contrast grammatical and ungrammatical structures, researchers have uncovered that while top-tier models like LLaMA-3-70B are highly competent, they still struggle with the complex, long-distance dependencies and unique case-marking systems inherent to the Urdu language.
The Challenge of Urdu Syntax
While LLMs demonstrate impressive performance across many languages, evaluation is often restricted to general reasoning or world knowledge. We know relatively little about whether these models truly "understand" the fine-grained grammatical rules of low-resource languages like Urdu.
Urdu presents unique linguistic challenges, including split ergativity (where subject marking changes based on verb aspect), complex honorifics, and oblique case marking. Existing benchmarks were either too small—such as MultiBLiMP’s 1,000 Urdu examples—or focused primarily on English. UrBLiMP addresses this gap by providing a rigorous framework to test if a model can distinguish a correct Urdu sentence from a subtly incorrect one.
Methodology: The Minimal Pairs Task
The authors adopted the Minimal Pairs methodology, popularized by the English BLiMP benchmark. In this zero-shot task, a model is presented with two sentences differing by only one grammatical feature (e.g., correct vs. incorrect verb agreement). The model "passes" if it assigns a higher probability (lower perplexity) to the grammatical version.
To construct the dataset, the team employed a hybrid pipeline:
- Template Extraction: Utilizing the Urdu Treebank to identify specific syntactic patterns.
- Large-Scale Mining: Using regular expressions to extract candidate sentences from a 735-million-token Urdu corpus.
- Controlled Perturbation: Systematically introducing errors, such as swapping masculine and feminine verb forms, to create ungrammatical pairs.
- Human Validation: 17 native speakers verified the pairs, achieving a 96.1% agreement rate.
Key Insights
- The "Instruction Tuning Tax": Interestingly, instruction-tuned models (marked "-it") often performed worse than their base pretrained counterparts ("-pt") on grammar. This suggests that fine-tuning for conversation might prioritize fluency over strict syntactic accuracy.
- Model Size vs. Performance: Smaller models like Gemma-3-12b-pt outperformed much larger models on specific tasks like dative object marking. This indicates that massive scale isn't always a prerequisite for capturing certain linguistic nuances.
- The Distance Problem: Models struggle significantly with "Long Distance Agreement." When multiple words separate a subject from its verb, the model's ability to maintain correct gender or number agreement drops sharply.
- Morphological Hurdles: Urdu’s "oblique" case marking remains a major weakness for most models, except for the very largest architectures.
Results and Performance
The benchmark results highlight a significant gap between current LLMs and human performance, though the best models are closing in.
- Human Baseline: 96.10% average accuracy.
- Top Performer: LLaMA-3-70B achieved the highest average accuracy at 94.73%, demonstrating that scale helps in capturing complex linguistic nuances.
- Efficiency Leader: Gemma-3-12b-pt reached 92.68%, proving to be highly effective for its size.
- Urdu-Specific Gains: Alif-1.0-8B, which underwent continued pretraining on Urdu, showed a massive improvement in handling Word Order Variation (97.03%) compared to the base LLaMA-3-8B (90.10%).
- Architecture Trends: Modern decoder-only LLMs significantly outperformed older encoder-only models like mBERT (78.77%) and mT5-small (50.69%).
Conclusion and Future Directions
UrBLiMP represents a significant step forward for Urdu NLP, providing a much-needed diagnostic tool for linguistic competence. While it covers 10 core phenomena, future iterations aim to include more complex structures like reciprocity and varied relative clauses. The findings suggest that "continued pretraining" on specialized corpora is a promising avenue for addressing specific weaknesses in long-distance agreement and case marking.
Summary updated