Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features

Luca Luciano Costanzo, Yashar Deldjoo, Maurizio Ferrari Dacrema, Markus Schedl, Paolo CremonesiarXiv 2019 · arXiv

TL;DR

This paper reveals a significant gap in how recommender systems understand us: the "implicit" profiles algorithms build by watching what we rate often bear little resemblance to our "explicit" tastes. By introducing a new dataset of direct user preferences for movie features, the authors show that while we can somewhat predict a user's favorite genres, our ability to implicitly capture their favorite actors and directors is surprisingly poor, with similarities as low as 9%.

What Problem Does This Paper Address?

Most modern recommender systems (RS) don't just look at what you bought; they look at why you might have bought it by analyzing item content—like movie genres, directors, or cast members. These systems build a user profile to estimate your interests.

However, there's a major methodological blind spot: researchers usually only evaluate the final recommendation list, not the accuracy of the user profile itself. We assume that if a user likes The Avengers, they must like Robert Downey Jr. and action movies. But is that true? Without a "ground truth" of what users actually say they like, we can't know if our profiling algorithms are capturing genuine taste or just statistical noise. This paper addresses this lack of evaluation by creating a benchmark for explicit feature preferences.

Methods

The authors compare two ways of building user profiles:

  1. Explicit Profiles ($p'_u$): Created directly from user input where they list their favorite genres, actors, and directors.
  2. Implicit Profiles ($p_u$): Inferred from the movies a user has rated using four different modeling techniques:
  3. Zhang Method: A simple binary approach—if a feature appears in a movie you liked, it's in your profile.
  4. Li Method: A frequency-based approach where features are weighted by how often they appear in your rated items.
  5. Symeonidis Method: A "user-centric" TF-IDF variant that weights rare features higher if they appear in fewer user profiles.
  6. TF-IDF Method: The authors' "item-centric" variant, weighting features based on their rarity across the entire movie catalog.

The core of their contribution is the evaluation framework shown in Figure 1, which measures the similarity (Cosine and Jaccard) between these two profile types to see how well the implicit models hold up against reality.

Figure 1: Main steps involved in a recommendation system leveraging content information, highlighting our contributions.

Datasets

  • PoliMovie: A new dataset created by the authors via a custom web application. It contains preferences from 194 users (155 deemed "reliable" via consistency tests) who explicitly selected favorite movies, genres, actors, and directors.
  • The Movies Dataset: A standard dataset of 45.3k movies used to provide the underlying metadata (features) for the items.
  • Comparison: Unlike MovieLens or the Netflix Prize dataset, PoliMovie provides the crucial "explicit feature ratings" needed to validate user profiling.

Key Insights and Ideas

  • The "Stan Lee" Noise: One of the most striking findings was that Stan Lee frequently appeared in implicit profiles as a top "preferred actor." Why? Because he has cameos in almost every Marvel movie. If a user likes Marvel movies, the algorithm assumes they love Stan Lee, even if they never explicitly listed him as a favorite. This highlights how implicit profiling can be "polluted" by frequent but non-defining features.
  • Genre is Easier, People are Harder: Algorithms are much better at guessing your favorite genres than your favorite actors or directors. This is likely because the "vocabulary" of genres is tiny (19 in this study) compared to the massive pool of actors (567k) and directors (58k).
  • Item-Centric Wins: Their proposed item-centric TF-IDF outperformed the existing user-centric Symeonidis method, suggesting that looking at feature rarity across the movie catalog is more informative than looking at rarity across user profiles.

Key Results

The results, summarized in Table 5, show that implicit profiling is far from perfect:

  • Genres: The best method (Li) achieved a 58.07% cosine similarity. While the highest in the study, it still leaves a massive 42% gap between inferred and actual taste.
  • Directors: Similarity dropped significantly to a maximum of 17.24%.
  • Actors: The results were the most dismal here, with a maximum similarity of only 9.13%.
Table 5: Average pairwise similarity between explicit and implicit user profiles, for all the methods.

These low numbers suggest that when we rate a movie, we are often only responding to a subset of its features. An implicit model that blindly absorbs every metadata tag associated with a liked movie is bound to be inaccurate.

Limitations and Future Work

The authors acknowledge that their user sample is relatively small (155 reliable users). Furthermore, the binary nature of the explicit ratings (you either list a favorite or you don't) might not capture the nuances of "disliking" certain features.

Future work aims to:

  • Generalize: Test these findings in other high-content domains like fashion, music, and tourism.
  • Noise Reduction: Develop more sophisticated profiling methods that can distinguish between "defining" features (the reason you liked the movie) and "incidental" features (like Stan Lee's cameos).
  • Feedback Loops: Investigate how to better integrate explicit feature feedback into standard recommendation pipelines to bridge this similarity gap.

Summary updated

Abstract

In order to improve the accuracy of recommendations, many recommender systems nowadays use side information beyond the user rating matrix, such as item content. These systems build user profiles as estimates of users' interest on content (e.g., movie genre, director or cast) and then evaluate the performance of the recommender system as a whole e.g., by their ability to recommend relevant and novel items to the target user. The user profile modelling stage, which is a key stage in content-driven RS is barely properly evaluated due to the lack of publicly available datasets that contain user preferences on content features of items. To raise awareness of this fact, we investigate differences between explicit user preferences and implicit user profiles. We create a dataset of explicit preferences towards content features of movies, which we release publicly. We then compare the collected explicit user feature preferences and implicit user profiles built via state-of-the-art user profiling models. Our results show a maximum average pairwise cosine similarity of 58.07\% between the explicit feature preferences and the implicit user profiles modelled by the best investigated profiling method and considering movies' genres only. For actors and directors, this maximum similarity is only 9.13\% and 17.24\%, respectively. This low similarity between explicit and implicit preference models encourages a more in-depth study to investigate and improve this important user profile modelling step, which will eventually translate into better recommendations.
BibTeX
@misc{costanzo2019towards,
  author = {Luca Luciano Costanzo and Yashar Deldjoo and Maurizio Ferrari Dacrema and Markus Schedl and Paolo Cremonesi},
  title = {{Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features}},
  year = {2019},
  eprint = {1908.11055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.11055},
}

Loading related content...

Figure