TL;DR
This paper investigates whether users can accurately assess recommended items in recommender system (RS) user studies without actually experiencing them. The authors found that for music, consumption significantly improved user assessments, while for movies, with richer textual descriptions, consumption had less impact, suggesting domain and information quality are crucial factors.
What Problem Does This Paper Address?
Recommender systems (RS) are ubiquitous, helping users discover new content. Evaluating these systems often relies on user studies where participants answer questionnaires about their experience. A common limitation is that participants rarely get to actually consume the recommended items (e.g., listen to a song, watch a movie) due to practical constraints. This raises a critical question: does this lack of direct experience skew user assessments of recommendation quality and overall user experience? If users are judging based on limited information like descriptions and images, are their evaluations truly representative of how they'd feel if they had experienced the item? This paper aims to quantify this impact, exploring if and when item consumption is necessary for valid user study results.
Methods
The researchers conducted two user studies, one focused on music recommendations and the other on movie recommendations. In both studies, participants were presented with recommended items and asked to complete questionnaires before and after they had the opportunity to consume the items.
- Music Study: Participants selected music genres, and the system recommended songs. They rated their satisfaction and completed a questionnaire (t1). Then, they listened to the songs for at least 30 seconds and completed a second questionnaire (t2).
- Movie Study: Participants selected a movie category, and the system recommended movies. They rated their satisfaction and completed a questionnaire (t1). Then, they watched one of the recommended movies and answered questions about their choice (t1). Subsequently, they watched the remaining movies and re-rated their satisfaction (t2 and t3).
The questionnaires used established instruments to measure aspects like perceived recommendation quality, diversity, novelty, information sufficiency, and overall satisfaction. Statistical analysis, specifically linear mixed-effect models, was used to compare assessments made before and after consumption, looking for significant differences.
Datasets
The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, the studies utilized:
- Music Study: The Spotify API was used to generate song recommendations based on user-selected genres.
- Movie Study: Short movies from a German newspaper's online article were used as recommended items.
Key Insights and Ideas
- Domain Matters: The impact of item consumption is highly domain-dependent. Music, being more abstract and emotionally driven, benefited significantly from actual listening, while movies, often supported by rich textual descriptions, showed less impact from consumption.
- Information Sufficiency is Key: When users have sufficient information before consumption (like detailed movie descriptions), they can often form accurate opinions without direct experience. Conversely, insufficient information (like basic song metadata) makes direct consumption crucial for a valid assessment.
- Consumption Bridges the Gap: For music, listening helped participants form stronger opinions, leading to higher satisfaction scores and a better understanding of recommendation quality. It also reduced choice difficulty and increased overall satisfaction.
- Typical User Studies Might Be Misleading: The findings suggest that user studies that don't allow item consumption might present an incomplete or even inaccurate picture of recommender system performance, especially in domains like music.
- Subjective Aspects Can Be Assessed Differently: While some aspects like perceived recommendation quality showed similar correlations before and after consumption in the movie study, other aspects like satisfaction and choice difficulty were more sensitive to consumption.
Key Results
- Music Study:
- Participants gave significantly higher ratings to recommendations after listening to songs.
- Perceived recommendation quality and overall satisfaction showed significant correlations with mean recommendation ratings both before and after listening.
- Listening had a significant effect on perceived information sufficiency, especially when initial information was low.
- Participants reported that listening was considered most influential afterwards, highlighting its subjective importance.
- Movie Study:
- Fewer significant interaction terms were found compared to the music study, indicating less impact of consumption.
- Mean ratings for individual recommendations showed a higher correlation between pre- and post-consumption assessments.
- Participants were often able to estimate well whether they would like recommended movies, likely due to rich textual descriptions.
- However, participants did settle on different movies more often when perceived recommendation quality was lower prior to consumption or when initial choice satisfaction was lower.
Limitations and Future Work
- Carryover Effects: The authors acknowledge that the influence of pre-consumption assessment on post-consumption results (carryover effects) is a potential limitation, though they attempted to control for it.
- Domain Specificity: While the studies covered music and movies, the generalizability to other domains (e.g., news articles, products) needs further investigation.
- Item Familiarity: The paper notes that prior familiarity with items could influence results, and this aspect warrants further study.
- Future Work: The authors suggest investigating the long-term stability of these findings, as people's memory of past events can be biased. They also emphasize the need for careful consideration of domain and information quality when designing user studies and avoiding direct comparisons across different experimental settings. Further research into the impact of item familiarity is also recommended.
Summary updated