Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation

Xi Wang, Hossein A. Rahmani, Jiqun Liu, Emine Yilmaz

Abstract

Conversational Recommendation System (CRS) is a rapidly growing research area that has gained significant attention alongside advancements in language modelling techniques. However, the current state of conversational recommendation faces numerous challenges due to its relative novelty and limited existing contributions. In this study, we delve into benchmark datasets for developing CRS models and address potential biases arising from the feedback loop inherent in multi-turn interactions, including selection bias and multiple popularity bias variants. Drawing inspiration from the success of generative data via using language models and data augmentation techniques, we present two novel strategies, 'Once-Aug' and 'PopNudge', to enhance model performance while mitigating biases. Through extensive experiments on ReDial and TG-ReDial benchmark datasets, we show a consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.

Introduction

Conversational Recommendation Systems (CRS) rely on multi-turn, natural language interactions to understand user preferences. However, they face a significant "feedback loop" problem. Selection bias occurs because training datasets are often sparse; for instance, only ~35% of items in the TG-ReDial dataset are ever mentioned. Popularity bias further skews results, as models tend to over-recommend "blockbuster" items, leading to a "Matthew Effect" where niche items are ignored. Additionally, dynamic user intent can shift during a conversation, requiring models to account for Cross-Episode Popularity (CEP) and User Intent-Oriented Popularity (UIOP).

Methodology

The researchers introduced a two-pronged approach involving bias formalization and LLM-powered data augmentation. They defined new metrics to quantify conversational bias, including Initial Item Coverage (IIC), CEP, and UIOP.

The workflow of the proposed data augmentation framework, illustrating the generation of synthetic data via GPT-3.5 and the subsequent training process.

Data Augmentation Strategies

The authors used GPT-3.5 to generate synthetic movie recommendation dialogues. Two strategies were tested:

  • Once-Aug (OA): Directly adding all synthetic dialogues to the training set.
  • PopNudge (PN): A strategic approach that identifies the items being discussed and then "nudges" the model by sampling synthetic dialogues featuring similar but less popular items. This forces the model to learn representations for the "long tail" of the dataset without destroying the natural distribution of human language.

Experimental Results

The strategies were tested on four state-of-the-art CRS models: ReDial, KBRD, TG-ReDial, and KGSF.

Comparative performance of augmentation strategies across multiple benchmarks, highlighting improvements in Hit@10, Hit@50, and reductions in Popularity Bias.

Accuracy and Coverage

PopNudge consistently improved Hit@10 and Hit@50 scores. For the base ReDial model, Hit@10 jumped from 0.0032 to 0.0220 when using PopNudge. Both augmentation strategies successfully increased Initial Item Coverage (IIC) to nearly 100% for both ReDial and TG-ReDial, effectively curing the selection bias.

Bias Mitigation

PopNudge significantly reduced CEP and UIOP scores, meaning the models became better at providing diverse recommendations that matched the specific "niche-ness" of user intent. The results suggest that strategic nudging is more effective than simple volume-based data augmentation.

Conclusion

This work highlights that selection bias is significantly more severe in conversational data than in traditional collaborative filtering. By using LLMs to strategically augment training data, the proposed PopNudge method effectively addresses the sparsity gap and popularity bias. Future directions include exploring multi-item synthetic dialogues and developing more robust evaluation frameworks for conversational quality.

Summary updated

Abstract

Conversational Recommendation System (CRS) is a rapidly growing research area that has gained significant attention alongside advancements in language modelling techniques. However, the current state of conversational recommendation faces numerous challenges due to its relative novelty and limited existing contributions. In this study, we delve into benchmark datasets for developing CRS models and address potential biases arising from the feedback loop inherent in multi-turn interactions, including selection bias and multiple popularity bias variants. Drawing inspiration from the success of generative data via using language models and data augmentation techniques, we present two novel strategies, 'Once-Aug' and 'PopNudge', to enhance model performance while mitigating biases. Through extensive experiments on ReDial and TG-ReDial benchmark datasets, we show a consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.
BibTeX
@inproceedings{wang2023improving,
  author = {Xi Wang and Hossein A. Rahmani and Jiqun Liu and Emine Yilmaz},
  title = {{Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=o7Cpy0nZZb},
}
Figure