Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, Jacki O'Neill

Introduction

As Large Language Models (LLMs) are increasingly deployed globally, standard benchmarks often fail to capture their performance in linguistically diverse, real-world environments. Most evaluations rely on curated, high-resource datasets like English Wikipedia, ignoring the fact that over 60% of the world is multilingual. In many regions, speakers frequently engage in code-mixing—the practice of switching between multiple languages within a single conversation. This paper investigates the "nuance gap" in LLMs by evaluating their ability to perform sentiment analysis on a unique dataset of code-mixed WhatsApp chats from Nairobi, Kenya, involving English, Swahili, and Sheng.

Methodology

The researchers employed a dual-method approach to assess seven leading models: GPT-4, GPT-4-Turbo, GPT-3.5-Turbo, Llama-2-70b, Mistral-7b, Mixtral-8x7b, and Gemma-7b.

  1. Quantitative Evaluation: Models were tested on a sentiment analysis task (Neutral, Positive, Negative) using weighted F1 scores.
  2. Qualitative HCI Analysis: To move beyond surface-level metrics, the researchers prompted models to provide justifications for their predictions. A team of native speakers evaluated 261 sampled justifications based on linguistic accuracy, contextual relevance, fluency, and transparency.

The WhatsApp Chat Dataset

The study utilized a dataset of 3,719 messages originally collected from a WhatsApp group supporting young people living with HIV in Nairobi. This data is characterized by "chat speak"—heavy use of emojis, abbreviations, and misspellings (e.g., "gud 9t"). Linguistically, it is a complex blend of English, Swahili, and Sheng, a dynamic Nairobi slang that incorporates elements of various local languages. This environment provides a rigorous test for a model's cultural and linguistic intelligence.

Quantitative Results: The F1 Illusion

Initial quantitative results suggested high performance for several models. As shown in Figure 1, Mistral-7b achieved the highest overall F1 score (~85%), followed by GPT-4 (~75%) and Mixtral-8x7b (~72%). Llama-2-70b performed significantly worse, with an F1 score of approximately 43%.

F1 Score performance across seven LLMs on the Nairobi WhatsApp dataset.

However, the researchers argue that these high scores can be deceptive. A model might arrive at the "correct" sentiment label through flawed reasoning or by ignoring non-English components of the text entirely.

Qualitative Analysis: The Nuance Gap

The core contribution of this work lies in the qualitative evaluation of model justifications. While Mistral-7b led in F1 scores, its explanations were often shallow or based on incorrect translations. In contrast, GPT-4 and GPT-4-Turbo demonstrated superior "transparency" and "linguistic accuracy," consistently translating non-English spans correctly.

Comparison of model justifications for a code-mixed message in English and Swahili.

As illustrated in Figure 2, even when models correctly identify a sentiment as "Neutral," their internal logic varies. For the message "Hi guys meeting yetu imekuewaje" (How was our meeting?), Mistral-7b incorrectly identified it as a "Swahili greeting," whereas GPT-4 correctly identified it as an inquiry about a meeting's status. This highlights the "right for the wrong reasons" phenomenon.

Key Insights and Discussion

  • English Bias: Models frequently anchored their predictions on English words, often ignoring Swahili or Sheng portions. This is a critical failure point, as negative sentiments in this dataset were more commonly expressed in Swahili.
  • Cultural Nuance: Advanced models still struggle with deep cultural context. For instance, a sarcastic Swahili phrase telling someone to "stay in school" was interpreted literally as neutral advice, missing the intended hostility.
  • Robustness to Noise: GPT-4 variants showed greater resilience to the "noisy" nature of chat data (typos and non-standard grammar), while smaller models often hallucinated meanings when encountering misspellings.

Conclusion

The study concludes that high F1 scores in low-resource, code-mixed scenarios do not necessarily equate to true linguistic or cultural understanding. There is a pressing need for evaluation benchmarks that target cultural nuance and for the integration of "cultural intelligence" into model training. Future work should expand this analysis to other geographic contexts and larger qualitative samples to further bridge the gap between metrics and meaning.

Summary updated

BibTeX
@misc{ochieng2024beyond,
  author = {Millicent Ochieng and Varun Gumma and Sunayana Sitaram and Jindong Wang and Vishrav Chaudhary and Keshet Ronen and Kalika Bali and Jacki O'Neill},
  title = {{Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios}},
  year = {2024},
  eprint = {2406.00343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00343},
}
Figure