Towards Building Voice-based Conversational Recommender Systems: Datasets, Potential Solutions, and Prospects

Xinghua Qu, Hongyang Liu, Zhu Sun, Xiang Yin, Yew Soon Ong, Lu Lu, Zejun Ma

Introduction

Traditional Conversational Recommender Systems (CRSs) are predominantly text-based (TCRSs). While effective, TCRSs present significant barriers for users with visual impairments, limited literacy, or those in "eyes-busy" environments like driving. Furthermore, text interfaces often suffer from sparse or inaccurate user profile data. This paper explores the transition to Voice-based Conversational Recommender Systems (VCRSs), arguing that voice is a more natural, efficient, and "honest" modality. A user's voice inherently carries metadata—such as age and gender—that is difficult to falsify and highly valuable for personalization.

Methodology

The authors propose a multi-stage pipeline to transform static interaction data into a rich voice-based conversational environment. This pipeline bridges the gap between raw user-item interactions and natural vocal dialogue.

The multi-stage pipeline for dataset selection, dialogue simulation, and speaker assignment.

The process begins with Backbone Selection, utilizing datasets like Coat (e-commerce) and MovieLens-1M. ChatGPT-Driven Dialogue generation then creates natural-sounding conversation templates, moving away from rigid robotic prompts. These dialogues are converted to audio using VITS, a state-of-the-art neural speech synthesis model, with speakers assigned from the Vctk dataset based on verified user metadata. Finally, a Multi-task Classification Module (MCM) extracts demographic features from the audio to inform a Voice Feature Integration Module (VFIM) for final recommendations.

Dataset Benchmarks

The study introduces the first public benchmarks for VCRS: VCRS-Coat and VCRS-ML-1M. Unlike previous datasets like ReDial or OpenDialKG, these include synthesized voice audio aligned with verified user metadata, allowing for the training of models that "hear" user characteristics.

Comparison of conversation styles across different datasets, highlighting the dialogue structure in the Coat dataset.

Key Insights and Results

A core finding is the "honesty" of voice. While users may skip demographic fields in forms, their vocal acoustics provide reliable signals. Interestingly, the study found that while humans excel at gender identification (95%+), they struggle with age detection (30-50% accuracy). The proposed MCM model significantly outperformed humans in age detection, identifying vocal patterns invisible to the human ear.

Empirical results demonstrate that side information is critical: removing age and gender features led to a 16.3% drop in recommendation performance (NDCG@10) on the Coat dataset. Furthermore, the synthesized audio achieved a high Mean Opinion Score (MOS) of 4.73/5.0, proving its viability for real-world application.

Conclusion and Future Work

This work lays the foundation for VCRSs by providing public datasets and a robust feature-fusion framework. Future research should address the limitations of simulated conversations by incorporating real-world human recordings and exploring end-to-end dynamic systems trained via reinforcement learning to handle environmental noise and multi-speaker scenarios.

Summary updated

Abstract

Conversational recommender systems (CRSs) have become crucial emerging research topics in the field of RSs, thanks to their natural advantages of explicitly acquiring user preferences via interactive conversations and revealing the reasons behind recommendations. However, the majority of current CRSs are text-based, which is less user-friendly and may pose challenges for certain users, such as those with visual impairments or limited writing and reading abilities. Therefore, for the first time, this paper investigates the potential of voice-based CRS (VCRSs) to revolutionize the way users interact with RSs in a natural, intuitive, convenient, and accessible fashion. To support such studies, we create two VCRSs benchmark datasets in the e-commerce and movie domains, after realizing the lack of such datasets through an exhaustive literature review. Specifically, we first empirically verify the benefits and necessity of creating such datasets. Thereafter, we convert the user-item interactions to text-based conversations through the ChatGPT-driven prompts for generating diverse and natural templates, and then synthesize the corresponding audios via the text-to-speech model. Meanwhile, a number of strategies are delicately designed to ensure the naturalness and high quality of voice conversations. On this basis, we further explore the potential solutions and point out possible directions to build end-to-end VCRSs by seamlessly extracting and integrating voice-based inputs, thus delivering performance-enhanced, self-explainable, and user-friendly VCRSs. Our study aims to establish the foundation and motivate further pioneering research in the emerging field of VCRSs. This aligns with the principles of explainable AI and AI for social good, viz., utilizing technology's potential to create a fair, sustainable, and just world.
BibTeX
@inproceedings{qu2023towards,
  author = {Xinghua Qu and Hongyang Liu and Zhu Sun and Xiang Yin and Yew Soon Ong and Lu Lu and Zejun Ma},
  title = {{Towards Building Voice-based Conversational Recommender Systems: Datasets, Potential Solutions, and Prospects}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591876},
}
Figure