REASONER: An Explainable Recommendation Dataset with Comprehensive Labeling Ground Truths

Xu Chen, Jingsen Zhang, Lei Wang, Quanyu Dai, Zhenhua Dong, Ruiming Tang, Rui Zhang, Li Chen, Xin Zhao, Ji-Rong WenNeurIPS 2023 · OpenReview

TL;DR

REASONER is a groundbreaking dataset for explainable recommendation that finally uses "ground truth" labels from real users rather than third-party annotators or proxy reviews. By capturing multi-modal feedback (text and video) across different psychological aspects—like what persuades a user to click versus what they find informative after watching—it provides a much-needed benchmark for building more transparent and satisfying AI systems.

What Problem Does This Paper Address?

In the world of recommender systems (like Netflix or Amazon), "explainability" is the holy grail. We don't just want to know what to recommend; we want to know why. However, existing datasets for this task have three major flaws:

  1. The Proxy Problem: Most datasets use user reviews as a "proxy" for explanations. But a review written after the fact isn't necessarily the reason a user clicked in the first place.
  2. The Annotator Gap: Many datasets use third-party workers to guess why a user liked an item. These workers aren't the actual users, so their "ground truth" is just a guess.
  3. The One-Size-Fits-All Fallacy: Most research treats "explanation" as a single goal. In reality, a user might need an explanation to be persuaded to try something new, but a different explanation to be informed about the item's actual features.
Persuasiveness vs. Informativeness in Recommendations

The authors introduce REASONER to bridge these gaps by collecting data from the actual users across multiple modalities and intent aspects.

Methods

The researchers took a "build it and they will come" approach. They developed a custom video recommendation platform and recruited approximately 3,000 labelers to use it.

The 5-Step Data Collection Workflow

The workflow was meticulously designed to capture the user's state of mind at different stages:

  • Pre-watching (Persuasiveness): Before watching a video, users were asked which features (tags or video previews) motivated them to click.
  • Post-watching (Informativeness & Satisfaction): After watching, users identified which features were most informative and which they were most satisfied with.
  • Multi-modality: The system provided both Tags (textual features processed from comments and metadata) and Previews (visual keyframes extracted from the video).
  • Personality Profiling: Users took a "Big Five" personality test, allowing researchers to correlate personality traits with explanation preferences.

To ensure high data quality, the team implemented strict filtering rules, including checks for "speed-running" the labeling, inconsistent answers, and manual audits by lab students.

Datasets

The paper introduces the REASONER dataset:

  • Scale: 2,997 users, 4,672 videos, and 6,115 unique tags.
  • Interactions: 58,497 total interactions.
  • Comparison: Unlike previous datasets (like Extra or ArgEx), REASONER is the first to check all the boxes: real-user labels, multi-aspect explanations, general applicability, and multi-modal content.

The authors also released a unified library containing implementations of several well-known explainable recommender models (like EFM, TriRank, and various "DERM" deep learning models) to serve as a standardized benchmarking suite.

Key Insights and Ideas

  • The "Aspect Gap": One of the most striking findings is that the Jaccard similarity between "Persuasiveness" (why I clicked) and "Informativeness" (what I learned) is low (around 0.35–0.45). This proves that a single explanation cannot satisfy all user needs; what gets someone in the door isn't necessarily what they find valuable once they're inside.
  • Modality Matters: About 38% of users prefer multi-modal explanations (text + visual). This suggests that models relying solely on text are missing a huge chunk of user preference.
  • Personality and Preference: By including Big Five data, the paper opens the door for "psychology-informed" recommendations—tailoring the style of an explanation to a user's specific personality type.

Key Results

The authors benchmarked 18 different models on the new dataset. Key findings include:

  • Performance Trade-offs: Models optimized for persuasiveness often performed poorly on informativeness. For example, the SULM model excelled at predicting persuasiveness tags but saw a significant drop when asked to predict what users found informative.
  • Deep Learning Edge: The "DERM" (Deep Explainable Recommender Model) family generally provided more stable and accurate results across different tasks compared to traditional matrix factorization methods.
  • Rating Prediction: Interestingly, the TRDM model achieved the best results in predicting actual user ratings (RMSE of 1.19), suggesting that better feature-based explanations correlate with better preference modeling.

Limitations and Future Work

While REASONER is a massive step forward, the authors acknowledge some limitations:

  • Domain Specificity: The dataset is currently focused on short videos. User behavior and explanation needs might differ significantly for news, music, or e-commerce.
  • Scale: While 3,000 users is excellent for an academic study, it is still small compared to the massive, noisy logs used by industrial giants.
  • Future Directions: The team plans to extend the dataset to other domains and explore "explanation fairness"—ensuring that AI explanations don't inadvertently discriminate based on user demographics or personality traits.

Summary updated

Abstract

Explainable recommendation has attracted much attention from the industry and academic communities. It has shown great potential to improve the recommendation persuasiveness, informativeness and user satisfaction. In the past few years, while a lot of promising explainable recommender models have been proposed, the datasets used to evaluate them still suffer from several limitations, for example, the explanation ground truths are not labeled by the real users, the explanations are mostly single-modal and around only one aspect. To bridge these gaps, in this paper, we build a new explainable recommendation dataset, which, to our knowledge, is the first contribution that provides a large amount of real user labeled multi-modal and multi-aspect explaination ground truths. In specific, we firstly develop a video recommendation platform, where a series of questions around the recommendation explainability are carefully designed. Then, we recruit about 3000 high-quality labelers with different backgrounds to use the system, and collect their behaviors and feedback to our questions. In this paper, we detail the construction process of our dataset and also provide extensive analysis on its characteristics. In addition, we develop a library, where ten well-known explainable recommender models are implemented in a unified framework. Based on this library, we build several benchmarks for different explainable recommendation tasks. At last, we present many new opportunities brought by our dataset, which are expected to promote the field of explainable recommendation. Our dataset, library and the related documents have been released at https://reasoner2023.github.io/.
BibTeX
@inproceedings{chen2023reasoner,
  author = {Xu Chen and Jingsen Zhang and Lei Wang and Quanyu Dai and Zhenhua Dong and Ruiming Tang and Rui Zhang and Li Chen and Xin Zhao and Ji-Rong Wen},
  title = {{REASONER: An Explainable Recommendation Dataset with Comprehensive Labeling Ground Truths}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=WMOPSuB9Cv},
}
Figure