Abstract
Conversational recommender systems (CRSs) are increasingly leveraging large language models (LLMs) to interpret user intent. However, these models often inherit and amplify societal biases present in their training data. This study investigates how neutral linguistic variations—such as names or locations—can trigger significant shifts in recommendation price and category, revealing deep-seated racial, gender, and socioeconomic biases.
Introduction
As AI assistants like Alexa and Google Assistant evolve, they rely more heavily on models like BERT to parse natural language. While effective, these models are often "pre-polluted" with intrinsic biases from their massive training corpora. This research explores the "unintended bias" problem: how variations in language that should be neutral (like a person's name or workplace) lead to significantly shifted price and category distributions, effectively creating a form of digital redlining.
Methodology: The LMRec Architecture
The researchers focused on a specific architecture called LMRec, which utilizes a fine-tuned BERT encoder paired with a recommendation decoder. To measure bias, they employed a template-based analysis where neutral sentences were populated with demographic-specific identifiers.
To quantify the results, the team developed two primary metrics:
- Price Percentage Score: Measuring the probability of specific price levels (from $ to $$$$) being recommended for a given demographic group.
- Association Score: A variation of the Word Embedding Association Test (WEAT) used to measure how strongly certain restaurant categories are linked to specific bias dimensions.
Data and Experimental Design
The study utilized the Yelp review dataset, spanning 12 years across seven North American cities. The resulting dataset included over 535,000 reviews for Atlanta alone, covering thousands of businesses. Demographic name lists were sourced from established sociological studies on labor market discrimination to ensure the substitutions reflected real-world social perceptions.
Key Insights: Stereotypes and Gatekeeping
The study revealed several critical ways bias manifests in conversational AI:
- Socioeconomic Gatekeeping: Mentioning high-status locations (e.g., "law office") triggered higher-priced suggestions, while mentioning a "mosque" or "convenience store" lowered the average price range.
- Stereotype Reinforcement: When a user mentioned a same-sex partner, the system significantly increased recommendations for "gay bars." While seemingly relevant, this represents a stereotype-driven inference not explicitly requested by the user.
- Intersectional Dominance: In tests combining race and gender, racial bias was encoded more strongly than gender bias. The "Black" name dimension caused a much larger shift in price distribution than the "Female" name dimension.
Results: Quantifying the Disparity
The most striking finding was the consistent racial price gap. Across all cities, there was a massive divergence at the lowest price level ($) based on name association.
Quantitatively, the percentage score for Black-associated names at the $ level was 0.6949, compared to 0.3051 for white-associated names. Furthermore, gendered steering was evident: male names were significantly more likely to receive recommendations for alcohol-serving establishments (pubs, breweries), while female names were steered toward desserts, bakeries, and cafes.
Conclusion and Future Directions
The high accuracy of LMRec in standard recommendation metrics masks its underlying unfairness, proving that accuracy is a poor proxy for fairness. The authors suggest that future work must explore Counterfactual Data Augmentation (CDA)—training models on swapped versions of the same text—and Fair Ranking algorithms to enforce category constraints. Ultimately, the paper serves as a warning that the language-handling power of LLMs comes with a high risk of automating and scaling human prejudice.
Summary updated