CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems

Nolwenn Bernard, Hideaki Joko, Faegheh Hasibi, Krisztian BalogarXiv 2024 · DOI · arXiv

TL;DR

CRS Arena is an open-source, crowdsourced benchmarking platform that evaluates Conversational Recommender Systems (CRSs) through side-by-side, anonymous human preference battles. By analyzing 474 real-user conversations, the researchers discovered a striking disconnect: traditional offline evaluation metrics (like Recall@10) negatively correlate with actual human satisfaction, highlighting the critical need for interactive, human-in-the-loop evaluation.

What Problem Does This Paper Address?

Evaluating Conversational Recommender Systems (CRSs) is notoriously difficult. Unlike traditional recommenders that output a static list of items, a CRS must engage in a multi-turn, natural language dialogue to uncover user preferences.

Currently, the field relies heavily on offline metrics (e.g., response fluency or recommendation accuracy on static datasets) or simulated user studies. These methods fail to capture the messy, unpredictable nature of real human interaction. While live user studies are the gold standard, they are traditionally expensive, time-consuming, and difficult to standardize across different research prototypes.

Methods

Inspired by the popular Chatbot Arena platform for LLMs, the authors developed CRS Arena. The platform introduces a scalable, gamified approach to human evaluation:

  1. Matchmaking: A "Battle Manager" pairs two anonymous CRSs for a side-by-side comparison, prioritizing systems with the fewest recorded conversations to ensure uniform data collection.
  2. Sequential Interaction: A user interacts with both systems independently to get a recommendation (e.g., asking for movie suggestions).
  3. Granular & Global Feedback: During the chat, users can explicitly flag turns with "Satisfied" or "Frustrated" buttons. Once both chats finish, the user votes on a winner or declares a tie.
  4. Elo Ranking: The platform uses these pairwise wins/losses to calculate an ongoing Elo rating for each system.
Overview of the main components of CRS Arena, including the Battle Manager and User Interface.

Unlike Chatbot Arena, where both models receive the exact same prompt, CRS Arena allows independent, multi-turn dialogues tailored to how each specific recommender steers the conversation.

Figure 2: Screenshot of the CRS Arena.

Datasets

The authors released CRSArena-Dial, a unique dataset containing 474 conversations annotated with turn-level human feedback and overall battle outcomes.

The data was collected across two distinct environments to test the platform's robustness:

  • Open Environment: Publicly accessible users on the Hugging Face Hub (254 conversations, 81 votes).
  • Closed Environment: Verified crowd workers via Prolific, who were instructed to interact at least 5 times per system to ensure deeper conversations (220 conversations, 104 votes).

The evaluated systems comprised 9 distinct configurations based on popular architectures (KBRD, BARCOR, UniCRS, ChatGPT, and CRB-CRS) trained on standard dialogue corpora like ReDial and OpenDialKG.

Key Insights and Ideas

  • The Offline Metric Illusion: The most shocking finding was a negative Spearman correlation ($\rho = -0.238$) between the models' traditional Recall@10 (R@10) scores and their human-voted Elo ratings. A system can be excellent at predicting the next item in a static dataset but highly frustrating to talk to in real life.
  • Low Overall User Satisfaction: Even the top-performing models (ChatGPT-based variants) only satisfied users roughly half the time (~52%). Common failure modes included a lack of understanding of user constraints, repetitive responses, and high latency.
  • Crowdsourcing is Highly Robust: The rankings and conversation characteristics (like response diversity and length) between the unpaid public ("open") and paid crowd workers ("closed") were highly correlated ($r = 0.763$ for Elo ratings). This proves that public crowdsourcing is a viable, low-cost method for scaling up CRS evaluation.

Key Results

The final leaderboard combined both open and closed environments to rank the 9 baseline systems.

Table 1: Preliminary ranking of systems and level of satisfaction across different environments. R@10 shows the recall of each CRS as reported in Wang et al. [18]. The rank position is shown in parentheses for Elo rating and R@10.

As shown in Table 1, ChatGPT_OpenDialKG and ChatGPT_ReDial dominated the arena, tying for the top spot with an Elo rating of 1102. Meanwhile, smaller, specialized research models like KBRD and UniCRS struggled significantly in live deployment, with KBRD_OpenDialKG hitting a rock-bottom user satisfaction rate of just 1.7%.

Limitations and Future Work

The authors note a few engineering and environmental limitations:

  • Hardware & Latency Constraints: The platform was deployed on modest hardware (2 CPU cores, 16 GB RAM), causing high latency for larger models, which negatively impacted user experience and satisfaction scores.
  • Data Noise: Because the open environment is unmoderated, the dataset contains some noisy or incomplete interactions.

Future Directions: The team plans to implement more sophisticated matchmaking algorithms (moving beyond simple uniform selection), scale up the hosting infrastructure to mitigate latency, and invite the broader AI community to integrate new conversational recommenders into the arena.

Summary updated

Abstract

We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.
BibTeX
@misc{bernard2024crs,
  author = {Nolwenn Bernard and Hideaki Joko and Faegheh Hasibi and Krisztian Balog},
  title = {{CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems}},
  year = {2024},
  eprint = {2412.10514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10514},
}
Figure