Beyond Metrics: Evaluating LLMs Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, Jacki O’Neill

Abstract: Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, Jacki O’Neill. Proceedings of the Sixth Workshop on African Natural Language Processing (AfricaNLP 2025).

2025.

Introduction

While Large Language Models (LLMs) have achieved remarkable milestones, their performance is predominantly measured against high-resource, standardized benchmarks. However, real-world communication is often messy and multilingual. Up to 60% of the global population is bilingual, and "code-mixing"—alternating between languages in a single conversation—is common. This paper evaluates seven popular LLMs on a real-world, code-mixed WhatsApp dataset from Kenya to determine if they truly understand cultural and linguistic nuances or if standard metrics like F1 scores are providing a false sense of progress.

Methodology

The researchers evaluated seven LLMs: GPT-4, GPT-4-Turbo, GPT-3.5-Turbo, Llama-2-70b, Mistral-7b, Mixtral-8x7b, and Gemma-7b. The core task was Sentiment Analysis (Positive, Negative, Neutral). Instead of simple labels, the models were prompted to provide a natural language justification (under 200 words) highlighting the specific text spans that influenced their decision.

The evaluation was two-pronged:

  1. Quantitative: Standard weighted F1 scores across the entire dataset.
  2. Qualitative: A deep, human evaluation of 261 sampled messages by native speakers, analyzing justifications for linguistic accuracy, cultural relevance, and transparency.

The Dataset: Authentic Kenyan WhatsApp Conversations

The study utilized a unique dataset of 3,719 messages from a health-focused peer support group in Nairobi, Kenya. The data features heavy code-mixing between English, Swahili, and Sheng (an urban slang). To preserve the reality of "chat speak," the authors did not normalize punctuation, emojis, or colloquial abbreviations.

Results: The Deception of Quantitative Metrics

Quantitative results showed that Mistral-7b achieved the highest overall F1 score, largely by dominating the majority "Neutral" class. However, the qualitative analysis revealed a significant discrepancy between score and actual understanding.

Figure 1: Overall F1 Scores (%) for the evaluated LLMs. While Mistral-7b leads quantitatively, qualitative analysis suggests this may not reflect true linguistic comprehension.

The "Aha" Moment: Right Answers for Wrong Reasons

The paper's most critical insight is that high F1 scores can be an illusion. Reading the justifications for Mistral-7b and Mixtral-8x7b revealed they frequently mistranslated Swahili and Sheng words. They often arrived at the correct sentiment label by latching onto English words while ignoring or misinterpreting the local language segments.

In contrast, GPT-4 and GPT-4-Turbo displayed much higher linguistic resilience, successfully navigating spelling errors and accurately translating code-mixed spans in their reasoning.

Figure 3: Qualitative comparison of model justifications for a code-mixed message. This illustrates how different models explain their reasoning, revealing varying levels of actual comprehension of Swahili and Sheng.

Key Findings and Cultural Blindspots

  • English Bias: Weaker models tended to ignore local language segments, missing crucial sentiment indicators often expressed in Swahili or Sheng.
  • Transparency: Models like Llama-2-70b and Mistral-7b often lacked transparency, failing to cite the specific text spans that informed their decisions.
  • Universal Cultural Gaps: Even advanced models like GPT-4 struggled with deep cultural subtleties. Metaphorical or culturally-specific dismissive phrases were often interpreted literally, missing the underlying social weight.

Conclusion

The authors advocate for a shift in how LLMs are evaluated for low-resource and culturally diverse contexts. By combining qualitative human-centric methods with standard NLP metrics, researchers can better identify whether a model truly understands the text or is merely exploiting statistical correlations. Future work should focus on integrating cultural intelligence into model training and refining the transparency of model explanations.

Summary updated

BibTeX
@misc{ochieng2024beyond,
  author = {Millicent Ochieng and Varun Gumma and Sunayana Sitaram and Jindong Wang and Vishrav Chaudhary and Keshet Ronen and Kalika Bali and Jacki O’Neill},
  title = {{Beyond Metrics: Evaluating LLMs Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios}},
  year = {2024},
  eprint = {2406.00343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00343},
}
Figure