Generalists vs. Specialists: Evaluating Large Language Models for Urdu

Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, Awais AtharEMNLP 2024 · DOI

Abstract

Urdu, despite having over 70 million native speakers, remains a low-resource language in the field of Natural Language Processing (NLP). This study evaluates the efficacy of large-scale "generalist" models (e.g., GPT-4-Turbo) versus task-specific "specialist" models (fine-tuned XLM-R, mT5, and Llama-3-8b) across 14 Urdu-centric tasks. Our findings reveal a distinct performance split: while fine-tuned specialists dominate structured classification benchmarks, generalist models are overwhelmingly preferred by human evaluators for generative tasks. This "Metric-Human Paradox" suggests that current automated metrics may fail to capture the nuances of natural Urdu, often rewarding models that mimic translation artifacts.

Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized NLP, yet their performance in non-English, low-resource languages remains inconsistent. For a language like Urdu, researchers face a fundamental architectural choice: deploy a massive, multi-purpose generalist model or fine-tune smaller, multilingual models for specific tasks. This research investigates this trade-off, evaluating models on classification and generation tasks while examining the alignment between automated metrics and human judgment.

Methodology

The evaluation framework compared two primary categories of models:

  1. Generalists: GPT-4-Turbo and base Llama-3-8b, evaluated in zero-shot and few-shot (3-shot, 6-shot) configurations, including Chain-of-Thought (CoT) prompting.
  2. Specialists: XLM-Roberta-large (classification), mT5-large (generation), and Llama-3-8b fine-tuned via Low-Rank Adaptation (LoRA) on Urdu-specific datasets.

The study covered 14 tasks, split evenly between classification (e.g., Sentiment Analysis, POS Tagging, NER) and generation (e.g., Summarization, Transliteration, AI Assistant).

Experimental Setup and Datasets

The researchers utilized a combination of native Urdu datasets and translated benchmarks. Classification tasks leveraged the Urdu IMDB dataset, Abuse Detection (Akhter et al.), and Universal Dependencies for POS tagging. Generative tasks used XSUMUrdu for summarization and the "UrduAssistant" dataset—comprising over 67,000 translated prompts—to evaluate conversational capabilities.

Results and Discussion

The Specialist Edge in Classification

In structured linguistic tasks, fine-tuning proved essential. The specialist Llama-FT model significantly outperformed generalist counterparts in identifying complex grammatical structures and named entities.

Confusion matrices comparing GPT-4 CoT and Llama-3-8B-FT on Named Entity Recognition (NER) tasks.

As shown in the NER confusion matrices (Figure 6), the fine-tuned specialist maintains higher precision across categories like Person, Organization, and Location compared to the GPT-4 CoT approach. Similarly, for Part-of-Speech (POS) tagging, the specialist model demonstrated robust performance.

Confusion matrix for the fine-tuned Llama-3-8B model on Part-of-Speech (POS) tagging.

Figure 5 illustrates the specialist's high accuracy in POS tagging, particularly in distinguishing between Nouns, Verbs, and Adjectives, where generalist models often struggle due to the morphological complexity of Urdu.

The Metric-Human Paradox in Generation

A critical finding of this study is the disconnect between automated metrics and human preference. In generative tasks like summarization and translation, specialist models like mT5 often achieved higher SacreBLEU scores. However, in side-by-side human evaluations, GPT-4-Turbo achieved a win rate exceeding 90%.

This discrepancy is attributed to "translation artifacts" in benchmarks. Many Urdu datasets are translated from English; automated metrics reward models that replicate the stiff, unnatural phrasing of these translations. Human evaluators, conversely, prefer the more natural and contextually aware outputs of larger generalist models, even if they deviate from the "ground truth" translation.

Conclusion

The study concludes that while fine-tuning remains the gold standard for structured classification in Urdu, the scale and reasoning capabilities of generalist models like GPT-4-Turbo provide a superior user experience for open-ended generation. To bridge this gap, the authors emphasize the need for high-quality, native Urdu datasets that move beyond simple translation, alongside the exploration of techniques like Retrieval-Augmented Generation (RAG) to provide generalist models with better linguistic context.

Summary updated

BibTeX
@inproceedings{arif2024generalists,
  author = {Samee Arif and Abdul Hameed Azeemi and Agha Ali Raza and Awais Athar},
  title = {{Generalists vs. Specialists: Evaluating Large Language Models for Urdu}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.426},
}
Figure