Abstract
Conversational Recommendation System (CRS) is a rapidly growing research area that has gained significant attention alongside advancements in language modelling techniques. However, the current state of conversational recommendation faces numerous challenges due to its relative novelty and limited existing contributions. In this study, we delve into benchmark datasets for developing CRS models and address potential biases arising from the feedback loop inherent in multi-turn interactions, including selection bias and multiple popularity bias variants. Drawing inspiration from the success of generative data via using language models and data augmentation techniques, we present two novel strategies, 'Once-Aug' and 'PopNudge', to enhance model performance while mitigating biases. Through extensive experiments on ReDial and TG-ReDial benchmark datasets, we show a consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.
Introduction
Conversational Recommendation Systems (CRS) represent a paradigm shift in how users discover content, moving from static filtering to dynamic, natural language interactions. Despite their potential, these systems are plagued by data scarcity and systemic biases. Unlike traditional recommendation engines that benefit from decades of user-item interaction data, CRS models must learn from relatively small, dialogue-heavy datasets. This scarcity often leads to a "cold start" problem that is exacerbated by inherent biases in human conversation, where a small subset of popular items dominates the discourse.
The Challenge of Bias in CRS
Research identifies two primary forms of bias that hinder the effectiveness of CRS models:
- Selection Bias: This occurs because human-generated datasets only cover a fraction of the available item universe. For example, while the MovieLens dataset covers nearly 100% of available items, the ReDial dataset covers only 72%, and TG-ReDial covers a mere 34%. This leaves the model blind to a vast "long tail" of potentially relevant items.
- Popularity Bias (The Matthew Effect): In a feedback loop, models tend to recommend items that are already popular, which in turn increases their popularity in the training data. This creates a self-reinforcing cycle where the system defaults to "safe" popular bets rather than tailoring suggestions to specific user intent.
To better quantify these issues, this study introduces two new metrics: Cross-Episode Popularity (CEP), which measures the independence of recommendations across different conversational episodes, and User Intent-Oriented Popularity (UIOP), which measures how well a recommendation's popularity aligns with the popularity of the user's actual target item.
Proposed Methodology: PopNudge and Once-Aug
To address these biases, the researchers leveraged Large Language Models (LLMs), specifically GPT-3.5, to generate synthetic movie recommendation dialogues via in-context learning. Two strategies were tested for integrating this synthetic data:
- Once-Aug: A straightforward approach where the entire synthetic dataset is added to the training pool at once. While helpful for increasing item coverage, this method can be unstable and does not always improve performance on complex, topic-heavy datasets.
- PopNudge (PN): The core innovation of this research. PopNudge performs selective, batch-level augmentation. During training, if a batch contains a highly popular item, the system "nudges" the model by injecting synthetic dialogues featuring similar but less popular items. This targeted intervention balances the distribution without losing the natural patterns of human dialogue.
Experimental Results
The effectiveness of these strategies was evaluated using four baseline models (ReDial, KGSF, KBRD, and TG-ReDial) across two major benchmarks.
Accuracy and Coverage
The application of PopNudge led to consistent improvements in recommendation accuracy. For the KBRD model on the ReDial dataset, the Hit@10 score improved from 0.0387 to 0.0514. Furthermore, the data augmentation strategies successfully achieved 100% Initial Item Coverage (IIC) for both datasets, effectively eliminating the selection bias present in the original training sets.
Bias Mitigation
The UIOP scores decreased significantly following the implementation of PopNudge. This indicates that the models became more sensitive to specific user intents, moving away from a reliance on global popularity and toward more personalized, diverse recommendations.
Discussion and Future Directions
The study demonstrates that LLMs can serve as powerful "data imputers," providing context-rich signals that traditional data augmentation methods lack. By treating popularity as a proxy for intent and strategically nudging the model toward the long-tail distribution, researchers can break the feedback loops that stifle diversity in CRS.
However, limitations remain. Current synthetic dialogues are relatively simple, typically focusing on a single item per interaction. Real-world conversations are often more complex, involving multiple comparisons and shifting topics. Future research will focus on increasing the naturalness of these synthetic interactions and testing the "nudging" framework in other domains, such as e-commerce and news recommendation, to ensure broader applicability.
Summary updated