Measures of Similarity in Memory-Based Collaborative Filtering Recommender System

Shalini Christabel Stephen, Hong Xie, Shri Rai2017 · DOI

TL;DR

This paper dives deep into the various similarity measures used in memory-based collaborative filtering recommender systems, highlighting their limitations, particularly concerning data sparsity and capturing a user's comprehensive interests. The authors propose a novel approach involving hierarchical product categorization to address these issues and improve recommendation quality.

What Problem Does This Paper Address?

Recommender systems (RS) are crucial for navigating the overwhelming amount of information in e-commerce. Collaborative Filtering (CF) is a popular technique that leverages user-item interactions to make personalized recommendations. However, a significant challenge in CF, especially memory-based approaches, is data sparsity. This means that the user-item rating matrix is often very sparse, with users rating only a small fraction of available items. Consequently, finding users or items with many co-rated items (essential for many similarity measures) becomes difficult, leading to unreliable recommendations. Furthermore, existing methods often focus on individual item ratings, failing to capture a user's broader interests across product categories, which can result in less satisfactory recommendations.

Methods

The paper provides a comprehensive review and comparison of various memory-based collaborative filtering similarity measures. Key methods discussed include:

  • Pearson Correlation Coefficient: A standard measure that calculates the linear relationship between users' ratings of co-rated items.
  • Adjusted Cosine Similarity: An improvement over cosine similarity that accounts for "grade inflation" by normalizing ratings based on a user's average rating.
  • Slope One Predictor: A simple and efficient item-based CF algorithm that predicts ratings based on the average difference between item ratings.
  • Vector Similarity (Cosine Similarity): Treats users/items as vectors and measures similarity based on the angle between them.
  • Jaccard Index: Used for comparing sets, particularly useful for binary or sparse data, focusing on the overlap of rated items.
  • Union-Based Similarity (Bhattacharyya Coefficient in CF - BCF): A novel approach that aims to measure similarity even when there are no co-rated items, by considering global rating distributions and local correlations.

The paper also introduces a proposed technique that addresses the identified limitations:

  • Hierarchical Product Categorization: Products are organized into a tree structure (categories and sub-categories).
  • Weight Assignment to Unrated Items: A user's rating on an item contributes to the "potential interest score" of unrated items, with the contribution inversely proportional to the distance in the hierarchy.
  • Combined Similarity: The proposed method combines the BCF measure (for item similarity) with this "potential interest score" to generate predictions. This aims to alleviate sparsity and capture a user's comprehensive interest.

Datasets

Not explicitly discussed in the paper. The authors refer to "real-world application datasets" and "sparse datasets" generally, and use a sample utility matrix (Table 1) to illustrate grade inflation.

Key Insights and Ideas

  • Data Sparsity is a Major Hurdle: The core problem is that sparse data severely limits the effectiveness of traditional similarity measures that rely on co-rated items.
  • Beyond Co-rated Items: The paper emphasizes the need for similarity measures that can work even without overlapping ratings, pointing towards methods like BCF.
  • Comprehensive Interest Matters: Simply looking at individual item ratings isn't enough; understanding a user's interest across broader categories is crucial for better recommendations.
  • Hierarchical Structure as a Solution: Organizing products hierarchically offers a promising way to infer user interest in unrated items by leveraging their ratings on related items.
  • Trade-offs in CF: There's a clear trade-off between user-based and item-based CF. User-based offers serendipity but can be less scalable, while item-based is more scalable but may recommend items too similar to what the user already likes.

Key Results

The paper primarily focuses on analyzing and comparing existing methods and proposing a new approach. Quantitative results are not presented in detail, but qualitative insights are:

  • Pearson Correlation and Adjusted Cosine: Found to be effective but still susceptible to sparsity.
  • Slope One Predictor: Highlighted for its simplicity, ease of implementation, and improved efficiency and scalability, especially for item-based CF.
  • BCF and Union-Based Similarity: Show promise in handling sparsity by not solely relying on co-rated items.
  • Proposed Hierarchical Approach: Hypothesized to alleviate sparsity and capture comprehensive user interest, leading to more reliable recommendations.

Limitations and Future Work

  • Limitations of Existing Methods:
    • Reliance on co-rated items leads to unreliability in sparse datasets.
  • Individual item ratings don't capture comprehensive user interests.
  • User-based CF can be computationally expensive.
  • Item-based CF may lack serendipity.
  • Cold-start problem for new users and items is implicitly a challenge.
  • Future Work (Implied by the Proposed Technique):
    • Further empirical evaluation of the proposed hierarchical categorization approach is needed to validate its effectiveness in improving recommendation accuracy and user satisfaction.
  • Investigating optimal ways to define and utilize the hierarchical structure for different product domains.
  • Exploring how to combine the proposed method with other CF techniques or hybrid approaches.

Summary updated

Abstract

Collaborative filtering (CF) technique in recommender systems (RS) is a well-known and popular technique that exploits relationships between users or items to make product recommendations to an active user. The effectiveness of existing memory based algorithms depend on the similarity measure that is used to identify nearest neighbours. However, similarity measures utilize only the ratings of co-rated items while computing the similarity between a pair of users or items. In most of the e-commerce applications, the rating matrix is too sparse since even active users of an online system tend to rate only a few items of the entire set of items. Therefore, co-rated items among users are even sparser. Moreover, the ratings a user gives an individual item tells us nothing about his comprehensive interest without which the generated recommendations may not be satisfactory to a user. In order to be able to address these issues, a comprehensive study is made of the various existing measures of similarity in a collaborative filtering recommender system (CFRS) and a hierarchical categorization of products has been proposed to understand the interest of a user in a wider scope so as to provide better recommendations as well as to alleviate data sparsity.
BibTeX
@misc{stephen2017measures,
  author = {Shalini Christabel Stephen and Hong Xie and Shri Rai},
  title = {{Measures of Similarity in Memory-Based Collaborative Filtering Recommender System}},
  year = {2017},
  doi = {10.1145/3092090.3092105},
}
Figure