Beyond Fine-Tuning: Evaluating Prompt Engineering and Prompt Tuning for Roman Urdu Hate Speech Detection

Toneema Zubair, Muhammad Junaid Asif, Faisal Kamiran, Hafiz Hassan Saeed, Rana Fayyaz AhmadarXiv 2026 · arXiv

TL;DR

Detecting hate speech in Roman Urdu is notoriously difficult due to non-standard spellings and frequent code-mixing with English. This paper systematically evaluates whether prompt engineering and prompt tuning can bridge the performance gap for this low-resource language without the heavy computational cost of full fine-tuning. The authors demonstrate that while Parameter-Efficient Fine-Tuning (PEFT) still holds the performance crown, a novel "mixed prompt" approach—combining manual templates with learnable soft tokens—significantly boosts accuracy for resource-constrained environments.

What Problem Does This Paper Address?

Hate speech detection (HSD) is a critical tool for online safety, but most models are optimized for high-resource languages like English. Roman Urdu (Urdu written in the Latin alphabet) presents a "perfect storm" of NLP challenges:

  1. Low Resource: A scarcity of high-quality, annotated datasets.
  2. Linguistic Chaos: No standardized spelling (e.g., "muhabbat" vs. "muhabat"), heavy use of slang, and phonetic variations.
  3. Code-Mixing: Users constantly switch between Urdu and English within a single sentence.

Traditional fine-tuning often fails to generalize well in these noisy, data-sparse environments. The authors investigate if "prompt-based learning"—which guides a model's behavior via instructions or minimal parameter updates—is a more viable path forward.

Methods

The researchers explored two primary branches of prompt-based adaptation:

  • Prompt Engineering (Inference-only): Using GPT-3.5, they compared Zero-Shot (providing only instructions) against Few-Shot (instructions plus 10 labeled examples) to see how much "in-context learning" helps.
  • Prompt Tuning (Parameter-efficient): Using multilingual BERT (mBERT), they tested three strategies:
  • Prefix Prompting: Placing the task-specific template at the end of the input.
  • Cloze Prompting: Formulating the task as a "fill-in-the-blank" problem.
  • Mixed Prompting (Proposed): A hybrid approach that combines fixed human-readable text with "soft tokens"—learnable mathematical embeddings that are optimized during training while the rest of the model remains frozen.

The overall workflow is visualized in Figure 1, showing the transition from raw social media text to final toxicity prediction.

![Fig.

  1. Proposed framework for Roman Urdu Hate Speech Detection (RU-HSD) using prompt engineering and prompt tuning strategies.](/figures/art_a215d6e8fbad4f2b9a6f9270894c43ea)

Datasets

The study utilizes the PURUTT (Parallel Urdu and Roman Urdu Toxicity) dataset, a benchmark containing 72,771 social media comments. The dataset is highly imbalanced (roughly 18% toxic, 82% non-toxic), reflecting the real-world distribution of online content. It specifically captures the informal grammar and semantic ambiguity inherent in Roman Urdu.

Key Insights and Ideas

  • The "Few-Shot" Leap: For LLMs like GPT-3.5, adding just a handful of examples (few-shot) isn't just a minor improvement; it’s a transformative one. It provides the necessary "anchor" for the model to navigate the phonetic inconsistencies of Roman Urdu.
  • Human-AI Hybridization: The Mixed Prompt strategy (Figure 3) proved that human intuition (fixed templates) and machine optimization (soft tokens) are better together. By allowing a small number of parameters to learn the specific "flavor" of Roman Urdu toxicity, the model outperformed purely manual prompting.
  • The Efficiency Trade-off: While PEFT (like LoRA) yields the highest F1 scores, prompt engineering requires zero training. This makes it the superior choice for rapid deployment or for researchers without access to high-end GPU clusters.

![Fig.

  1. Mixed Prompt Tuning Architecture with Learnable Soft Tokens](/figures/art_08f7f14de7f44ddfabb18971433ca984)

Key Results

  • GPT-3.5 Performance: Few-shot prompting achieved an F1-score of 0.8565, a massive jump from the 0.6698 seen in zero-shot settings. This is clearly illustrated in the comparative metrics of Figure 4.
  • Prompt Tuning: Among the mBERT strategies, the Mixed Prompt achieved the highest F1-score (0.6852), outperforming both Prefix and Cloze formulations.
  • The Leaderboard: When compared against previously reported PEFT results, Mistral-7B with LoRA remains the top performer (F1: 0.9387). However, GPT-3.5 few-shot prompting is surprisingly competitive, outperforming mBERT-based PEFT.

![Fig.

  1. Comparative performance of GPT-3.5 using zero-shot and few-shot prompting across accuracy, precision, recall, and F1 score.](/figures/art_6a1d64865d5941ebb55b3269c092b973)

![Fig.

  1. F1 score comparison of prompting strategies and PEFT-based models](/figures/art_2c35641091a2424cbf653fc572db1342)

Limitations and Future Work

The authors acknowledge that while prompt tuning is efficient, it still lags behind PEFT methods like LoRA in raw predictive power for this specific task. Furthermore, the performance of Cloze prompting was inconsistent as the number of examples (K-value) increased, suggesting that template sensitivity remains a hurdle.

Future research directions include:

  • Investigating cross-domain generalization (e.g., training on Twitter data and testing on YouTube comments).
  • Applying these optimized prompting strategies to other low-resource South Asian languages.
  • Refining prompt optimization to reduce the sensitivity to specific wording in the templates.

Summary updated

Abstract

Low Resource Languages (LRLs) pose significant challenges for hate speech detection because of a scarcity of readily available annotated data, language variation, non-standard spelling, and context-dependent ambiguity. Beyond the problem of informal conventions of writing in Roman Urdu, there is also the issue of code-mixing. After working on the previous project about zero-shot learning and parameter-efficient fine-tuning (PEFT), this research systematically compares prompt engineering and prompt tuning for Roman Urdu hate speech detection. To study the effect of in-context examples, we will compare the zero-shot and few-shot prompting approaches. To understand the advantage of structured prompting and learnable prompt representations, we will conduct studies of manual vs. mixed prompt-based adaptation strategies. Experimentations are carried out on the PURUT dataset (~72,771 social media comments) with GPT-3.5, LLaMA-3, Mistral-7B, and multilingual BERT in several settings. Experimental results show that few-shot prompting always outperforms zero-shot prompting, with the Mistral-7B model scoring 0.67 F1 compared to 0.56 of the zero-shot models. Based on this observation, prompt-based adaptation is proposed to introduce additional improvements, with the proposed mixed prompt configuration able to yield a F1-score up to 0.87 which further surpasses the performance of regular manual prompts. Despite the larger size of the model, its performance trade-off between inference-based prompting, prompt optimization, and parameter-efficient model adaptation is compared to the previously reported PEFT-based F1-score, which is ~0.93.
BibTeX
@misc{zubair2026beyond,
  author = {Toneema Zubair and Muhammad Junaid Asif and Faisal Kamiran and Hafiz Hassan Saeed and Rana Fayyaz Ahmad},
  title = {{Beyond Fine-Tuning: Evaluating Prompt Engineering and Prompt Tuning for Roman Urdu Hate Speech Detection}},
  year = {2026},
  eprint = {2611.04342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04342},
}
Figure