Unintended Bias in Language Model-driven Conversational Recommendation

Tianshu Shen, Jiaru Li, Mohamed Reda Bouadjenek, Zheda Mai, Scott SannerarXiv 2022 · arXiv

Abstract

Conversational recommender systems (CRSs) are increasingly leveraging large language models (LLMs) to interpret user intent. However, these models often inherit and amplify societal biases present in their training data. This study investigates how neutral linguistic variations—such as names or locations—can trigger significant shifts in recommendation price and category, revealing deep-seated racial, gender, and socioeconomic biases.

Introduction

As AI assistants like Alexa and Google Assistant evolve, they rely more heavily on models like BERT to parse natural language. While effective, these models are often "pre-polluted" with intrinsic biases from their massive training corpora. This research explores the "unintended bias" problem: how variations in language that should be neutral (like a person's name or workplace) lead to significantly shifted price and category distributions, effectively creating a form of digital redlining.

Methodology: The LMRec Architecture

The researchers focused on a specific architecture called LMRec, which utilizes a fine-tuned BERT encoder paired with a recommendation decoder. To measure bias, they employed a template-based analysis where neutral sentences were populated with demographic-specific identifiers.

The LMRec architecture processes natural language queries through a BERT encoder to generate restaurant recommendations.

To quantify the results, the team developed two primary metrics:

  1. Price Percentage Score: Measuring the probability of specific price levels (from $ to $$$$) being recommended for a given demographic group.
  2. Association Score: A variation of the Word Embedding Association Test (WEAT) used to measure how strongly certain restaurant categories are linked to specific bias dimensions.

Data and Experimental Design

The study utilized the Yelp review dataset, spanning 12 years across seven North American cities. The resulting dataset included over 535,000 reviews for Atlanta alone, covering thousands of businesses. Demographic name lists were sourced from established sociological studies on labor market discrimination to ensure the substitutions reflected real-world social perceptions.

Key Insights: Stereotypes and Gatekeeping

The study revealed several critical ways bias manifests in conversational AI:

  • Socioeconomic Gatekeeping: Mentioning high-status locations (e.g., "law office") triggered higher-priced suggestions, while mentioning a "mosque" or "convenience store" lowered the average price range.
  • Stereotype Reinforcement: When a user mentioned a same-sex partner, the system significantly increased recommendations for "gay bars." While seemingly relevant, this represents a stereotype-driven inference not explicitly requested by the user.
  • Intersectional Dominance: In tests combining race and gender, racial bias was encoded more strongly than gender bias. The "Black" name dimension caused a much larger shift in price distribution than the "Female" name dimension.

Results: Quantifying the Disparity

The most striking finding was the consistent racial price gap. Across all cities, there was a massive divergence at the lowest price level ($) based on name association.

Price distribution comparison showing the significant disparity in recommendations at the lowest price level ($1.0) between demographic groups.

Quantitatively, the percentage score for Black-associated names at the $ level was 0.6949, compared to 0.3051 for white-associated names. Furthermore, gendered steering was evident: male names were significantly more likely to receive recommendations for alcohol-serving establishments (pubs, breweries), while female names were steered toward desserts, bakeries, and cafes.

Conclusion and Future Directions

The high accuracy of LMRec in standard recommendation metrics masks its underlying unfairness, proving that accuracy is a poor proxy for fairness. The authors suggest that future work must explore Counterfactual Data Augmentation (CDA)—training models on swapped versions of the same text—and Fair Ranking algorithms to enforce category constraints. Ultimately, the paper serves as a warning that the language-handling power of LLMs comes with a high risk of automating and scaling human prejudice.

Summary updated

Abstract

Conversational Recommendation Systems (CRSs) have recently started to leverage pretrained language models (LM) such as BERT for their ability to semantically interpret a wide range of preference statement variations. However, pretrained LMs are well-known to be prone to intrinsic biases in their training data, which may be exacerbated by biases embedded in domain-specific language data(e.g., user reviews) used to fine-tune LMs for CRSs. We study a recently introduced LM-driven recommendation backbone (termed LMRec) of a CRS to investigate how unintended bias i.e., language variations such as name references or indirect indicators of sexual orientation or location that should not affect recommendations manifests in significantly shifted price and category distributions of restaurant recommendations. The alarming results we observe strongly indicate that LMRec has learned to reinforce harmful stereotypes through its recommendations. For example, offhand mention of names associated with the black community significantly lowers the price distribution of recommended restaurants, while offhand mentions of common male-associated names lead to an increase in recommended alcohol-serving establishments. These and many related results presented in this work raise a red flag that advances in the language handling capability of LM-drivenCRSs do not come without significant challenges related to mitigating unintended bias in future deployed CRS assistants with a potential reach of hundreds of millions of end-users.
BibTeX
@misc{shen2022unintended,
  author = {Tianshu Shen and Jiaru Li and Mohamed Reda Bouadjenek and Zheda Mai and Scott Sanner},
  title = {{Unintended Bias in Language Model-driven Conversational Recommendation}},
  year = {2022},
  eprint = {2201.06224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.06224},
}
Figure