Abstract
Conversational Recommender Systems (CRS) aim to provide personalized item suggestions alongside natural language explanations. However, Large Language Models (LLMs) used in CRS often suffer from hallucinations, providing factually incorrect information. While Retrieval-Augmented Generation (RAG) can mitigate this, its application in CRS is hindered by preference misalignment and noisy training data. We present ESPRESSO, a framework that enhances passage retrieval through adaptive item selection and relevance-based groupwise learning (RGL). ESPRESSO dynamically adjusts the number of items for which information is retrieved based on recommendation confidence and employs a noise-robust training strategy. Experimental results on DuRecDial2.0 and KoRecDial datasets demonstrate that ESPRESSO significantly improves retrieval accuracy and generation quality, reducing hallucinations and providing more informative responses.
Introduction
Conversational Recommender Systems (CRS) are designed to simulate a helpful human assistant, providing both item recommendations and the reasoning behind them. Despite the power of Large Language Models (LLMs) in generating natural responses, they are prone to "hallucinations"—the generation of factually incorrect details. Integrating Retrieval-Augmented Generation (RAG) is a common solution, but it faces unique challenges in the CRS context: preference misalignment (retrieving facts for the wrong item due to recommendation uncertainty) and the high cost/noise of labeling relevant passages for training.
The ESPRESSO Framework
To address these challenges, we introduce ESPRESSO (Enhanced paSsage retrieval apPRoach via adaptivE item Selection and relevance-baSed grOupwise learning). ESPRESSO is designed to bridge the gap between recommendation and factual generation by making the retrieval process more robust to both system uncertainty and training noise.
The framework consists of four primary components: a Recommendation Module to identify user preferences, a Labeling Module to generate training data, a Passage Retrieval Module to fetch factual evidence, and a Generation Module to produce the final response.
Methods: Adaptive Selection and Groupwise Learning
ESPRESSO introduces two key innovations to improve CRS performance:
1. Adaptive Item Selection
Standard RAG systems typically retrieve information for only the top-ranked item. However, if a recommender is uncertain (e.g., 40% confidence for item A vs. 39% for item B), this "top-1" approach is brittle. ESPRESSO employs an adaptive selection strategy: it retrieves passages for a variable number of items until a cumulative confidence threshold ($\sigma_{conf}$) is reached. This ensures that the correct item's information is likely included in the retrieved set even if it isn't the absolute top recommendation.
2. Relevance-based Groupwise Learning (RGL)
Training a retriever for CRS is difficult because high-quality human labels for "relevant passages" are scarce. ESPRESSO uses a Labeling Module (combining BM25 and GPT-4o) to create pseudo-labels. To mitigate the noise in these automated labels, RGL groups passages by their predicted relevance. Instead of a standard pairwise contrastive loss, RGL optimizes the model to ensure that the average score of a group of likely-relevant passages is higher than that of irrelevant ones, providing a more stable training signal.
Experimental Results
The effectiveness of ESPRESSO was evaluated on the DuRecDial2.0 (English) and KoRecDial (Korean) datasets.
- Retrieval Performance: ESPRESSO consistently outperformed state-of-the-art retrievers like DPR, Contriever, and CoT-MAE. On DuRecDial2.0, it achieved a Hit@3 of 0.675, compared to 0.648 for standard top-1 retrieval methods.
- Generation Quality: Responses generated using ESPRESSO-retrieved passages showed significant improvements in BLEU scores. For instance, LLaMa2 with ESPRESSO reached a BLEU4 of 0.103, outperforming standard RAG baselines.
- Factuality: Qualitative analysis confirmed a substantial reduction in hallucinations. The system successfully included specific, accurate details such as directors and awards, rather than generic praise.
Conclusion
ESPRESSO provides a robust solution for factual conversational recommendation. By treating recommendation uncertainty as a parameter for retrieval and employing a groupwise learning strategy to handle noisy labels, it significantly enhances the reliability and informativeness of CRS. Future work will focus on reducing the computational cost of the labeling module and further refining the generator's ability to handle noise in expanded retrieval sets.
Summary updated