Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems

Abdulla Alshabanah, Murali AnnavaramEMNLP 2025

TL;DR

This paper reveals significant fairness disparities in recommendation systems for Arabic speakers, showing that users of certain dialects receive lower-quality recommendations. By leveraging recent NLP advancements in dialect identification, the study highlights how linguistic diversity can lead to unequal access to content, underscoring the need for more inclusive recommendation system design.

What Problem Does This Paper Address?

Recommendation systems are crucial for personalizing user experiences and driving engagement on digital platforms. However, these systems can exhibit bias, leading to unfair performance across different user groups. While research has explored various sources of bias, the linguistic diversity within large language communities, such as Arabic speakers, has been largely overlooked. Arabic, spoken by over 420 million people, encompasses numerous dialects that differ significantly in grammar and vocabulary, despite sharing a common root in Modern Standard Arabic (MSA). This dialectal variation creates a form of diglossia, where MSA is the formal language, while regional dialects are less represented in digital contexts. Previous research has primarily focused on identifying these dialects using NLP, but their impact on recommendation system fairness has remained largely unexamined. This paper aims to bridge this gap by investigating whether recommendation quality varies based on the Arabic dialect used by users, potentially leading to access disparities.

Methods

The researchers employed two main approaches:

  • Arabic Dialect Identification (ADI): They utilized a BERT-based model, previously developed by Keleg and Magdy (2023), to predict the most common dialect for each user based on their written reviews. This model classifies text into specific country-level dialects and also uses the Arabic Level of Dialectness (ALDi) score to distinguish between MSA and dialectal Arabic.
  • Recommendation System Modeling: They experimented with two popular recommendation system architectures:
  • Two-Tower Neural Network (TTNN): A standard architecture that uses separate neural networks for user and item features to generate dense representations.
  • DeepFM: A model that combines factorization machines (FM) with deep learning to capture both low-order and high-order feature interactions.

Both models were trained using a max-margin ranking loss to ensure positive items receive higher scores than negative ones.

Datasets

The study utilized two publicly available benchmark datasets:

  • Book Reviews in Arabic Dataset (BRAD): This dataset contains book reviews written in Arabic.
  • Large-scale Arabic Book Reviews Dataset (LABR): Another dataset of Arabic book reviews.

Both datasets were preprocessed into an implicit feedback format, and a leave-one-out strategy was used for splitting data into training, validation, and test sets.

Key Insights and Ideas

  • Dialect Matters for Fairness: The most striking insight is that the dialect a user speaks directly impacts the quality of recommendations they receive. This challenges the assumption that all Arabic speakers are treated equally by recommendation systems.
  • NLP as a Fairness Revealer: Advancements in NLP, specifically in dialect identification, are crucial tools for uncovering hidden biases in downstream applications like recommendation systems.
  • Multi-level Disparities: Fairness gaps are observed not only at the broad MSA vs. dialectal Arabic level but also at regional and fine-grained country-level dialect groupings, indicating that the problem is pervasive.
  • Model Sensitivity: While disparities exist across models, the behavior and extent of these differences can vary, suggesting that model architecture and training choices can influence fairness outcomes.

Key Results

The study found consistent patterns of fairness disparities across both datasets and models:

  • MSA vs. Dialectal Arabic: In BRAD, MSA users generally received better recommendations. In LABR, the gap was more pronounced, with DeepFM showing a notable difference in HR@10 scores between dialectal and MSA users.
  • Regional and Country-Level Differences: At the regional level (Nile, Gulf, Levant, Maghreb), specific user groups consistently received better performance. At the country level, disparities were even more pronounced, with some countries showing high HR@10 values while others had substantially lower scores. For instance, in BRAD, countries like Saudi Arabia, Oman, and Palestine showed significantly lower scores.
  • Quantifying Fairness Gaps: The User-oriented Group Fairness (UGF) metric revealed that fairness gaps increase as user groupings become finer-grained (from MSA vs. Dialectal to country-level dialects). For example, the UGF for HR@10 at the country-level was significantly higher than at the MSA vs. Dialectal level.

Limitations and Future Work

  • Limitations:
    • The accuracy of the dialect identification models used is a limiting factor.
  • The study is limited to written user reviews, which may not fully capture the nuances of dialect use in spoken or informal communication.
  • The paper does not propose solutions for mitigating these fairness issues, leaving that for future work.
  • Future Work: The authors suggest that future research should focus on developing fairness mitigation strategies to address the identified disparities and explore how these issues manifest in other contexts beyond written reviews.

Summary updated

BibTeX
@inproceedings{alshabanah2025mind,
  author = {Abdulla Alshabanah and Murali Annavaram},
  title = {{Mind the Dialect: NLP Advancements Uncover Fairness Disparities for Arabic Users in Recommendation Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure