TL;DR
REASONER is a groundbreaking dataset for explainable recommendation that finally uses "ground truth" labels from real users rather than third-party annotators or proxy reviews. By capturing multi-modal feedback (text and video) across different psychological aspects—like what persuades a user to click versus what they find informative after watching—it provides a much-needed benchmark for building more transparent and satisfying AI systems.
What Problem Does This Paper Address?
In the world of recommender systems (like Netflix or Amazon), "explainability" is the holy grail. We don't just want to know what to recommend; we want to know why. However, existing datasets for this task have three major flaws:
- The Proxy Problem: Most datasets use user reviews as a "proxy" for explanations. But a review written after the fact isn't necessarily the reason a user clicked in the first place.
- The Annotator Gap: Many datasets use third-party workers to guess why a user liked an item. These workers aren't the actual users, so their "ground truth" is just a guess.
- The One-Size-Fits-All Fallacy: Most research treats "explanation" as a single goal. In reality, a user might need an explanation to be persuaded to try something new, but a different explanation to be informed about the item's actual features.
The authors introduce REASONER to bridge these gaps by collecting data from the actual users across multiple modalities and intent aspects.
Methods
The researchers took a "build it and they will come" approach. They developed a custom video recommendation platform and recruited approximately 3,000 labelers to use it.
The workflow was meticulously designed to capture the user's state of mind at different stages:
- Pre-watching (Persuasiveness): Before watching a video, users were asked which features (tags or video previews) motivated them to click.
- Post-watching (Informativeness & Satisfaction): After watching, users identified which features were most informative and which they were most satisfied with.
- Multi-modality: The system provided both Tags (textual features processed from comments and metadata) and Previews (visual keyframes extracted from the video).
- Personality Profiling: Users took a "Big Five" personality test, allowing researchers to correlate personality traits with explanation preferences.
To ensure high data quality, the team implemented strict filtering rules, including checks for "speed-running" the labeling, inconsistent answers, and manual audits by lab students.
Datasets
The paper introduces the REASONER dataset:
- Scale: 2,997 users, 4,672 videos, and 6,115 unique tags.
- Interactions: 58,497 total interactions.
- Comparison: Unlike previous datasets (like Extra or ArgEx), REASONER is the first to check all the boxes: real-user labels, multi-aspect explanations, general applicability, and multi-modal content.
The authors also released a unified library containing implementations of several well-known explainable recommender models (like EFM, TriRank, and various "DERM" deep learning models) to serve as a standardized benchmarking suite.
Key Insights and Ideas
- The "Aspect Gap": One of the most striking findings is that the Jaccard similarity between "Persuasiveness" (why I clicked) and "Informativeness" (what I learned) is low (around 0.35–0.45). This proves that a single explanation cannot satisfy all user needs; what gets someone in the door isn't necessarily what they find valuable once they're inside.
- Modality Matters: About 38% of users prefer multi-modal explanations (text + visual). This suggests that models relying solely on text are missing a huge chunk of user preference.
- Personality and Preference: By including Big Five data, the paper opens the door for "psychology-informed" recommendations—tailoring the style of an explanation to a user's specific personality type.
Key Results
The authors benchmarked 18 different models on the new dataset. Key findings include:
- Performance Trade-offs: Models optimized for persuasiveness often performed poorly on informativeness. For example, the SULM model excelled at predicting persuasiveness tags but saw a significant drop when asked to predict what users found informative.
- Deep Learning Edge: The "DERM" (Deep Explainable Recommender Model) family generally provided more stable and accurate results across different tasks compared to traditional matrix factorization methods.
- Rating Prediction: Interestingly, the TRDM model achieved the best results in predicting actual user ratings (RMSE of 1.19), suggesting that better feature-based explanations correlate with better preference modeling.
Limitations and Future Work
While REASONER is a massive step forward, the authors acknowledge some limitations:
- Domain Specificity: The dataset is currently focused on short videos. User behavior and explanation needs might differ significantly for news, music, or e-commerce.
- Scale: While 3,000 users is excellent for an academic study, it is still small compared to the massive, noisy logs used by industrial giants.
- Future Directions: The team plans to extend the dataset to other domains and explore "explanation fairness"—ensuring that AI explanations don't inadvertently discriminate based on user demographics or personality traits.
Summary updated