Learning the Structure of Auto-Encoding Recommenders

Farhan Khawar, Leonard K. M. Poon, Nevin L. ZhangWWW 2020 · OpenReview

TL;DR

This paper introduces a novel "structure learning" approach for autoencoder-based recommender systems. By leveraging inherent item group relationships, they design a sparse and wide neural network architecture that significantly outperforms existing methods, especially in sparse data scenarios and cold-start situations, while using comparable computational resources.

What Problem Does This Paper Address?

Recommender systems often rely on collaborative filtering (CF) to suggest items based on user behavior. Autoencoders have emerged as powerful tools for CF, capable of modeling complex, non-linear relationships between users and items. However, current autoencoder recommenders typically use fully-connected (FC) neural networks. This "one-size-fits-all" approach can be inefficient, particularly with sparse data, leading to suboptimal generalization and performance. The core problem is that FC networks don't exploit the inherent structure of item relationships (e.g., items belonging to similar groups), leading to unnecessary connections and potentially noisy learning.

Methods

The authors propose a two-stage method to learn a sparse and wide autoencoder structure (SW-DAE).

  1. Structure Learning (Item Grouping):
    • Item Embeddings: They first learn low-dimensional embeddings for items. This is achieved by constructing an item-item co-occurrence graph from the user-item interaction matrix, forming a graph Laplacian, and then computing its eigenvectors. This process aims to capture item relationships while downplaying the influence of highly popular items.
  2. Overlapping Clustering: Using these item embeddings, they apply K-means to identify item clusters. Crucially, they allow items to belong to multiple clusters (overlapping groups), reflecting real-world scenarios where an item can have multiple facets (e.g., a movie being both action and drama).
  3. Connectivity Structure: Based on these overlapping item groups, they define the connectivity between the input and hidden layers of the autoencoder. Instead of a fully-connected layer, each neuron in the hidden layer corresponds to an item group. This neuron then connects to all items within its group, creating a sparse and wide architecture. The same structure is inverted for the decoder.
  4. Parameter Learning:
    • Once the sparse structure is defined, they train a denoising autoencoder (DAE) with this specific connectivity. They use standard stochastic gradient descent with a denoising criterion.

Datasets

The authors evaluated their approach on three publicly available datasets:

  • Movielens20M (ML20M): A dataset of user movie ratings, binarized for the study.
  • Netflix: Another movie rating dataset from the Netflix Prize, also binarized.
  • Million Song Dataset (MSD): A dataset of user song listening counts, binarized.

They used the same data splits and preprocessing as a prominent baseline (MULT-VAE) for direct comparison.

Key Insights and Ideas

  • Exploiting Item Structure is Key: The fundamental insight is that items are not independent. By explicitly modeling item group relationships, the autoencoder can learn more efficiently and effectively.
  • Sparse and Wide vs. Dense and Narrow: The proposed SW-DAE architecture is both sparse (fewer connections per neuron) and wide (more neurons, each representing a group). This contrasts with typical FC networks which are dense and can be narrow or wide but still fully connected.
  • Lower Spectral Norm, Better Generalization: The learned sparse structure leads to a significantly lower spectral norm of the weight matrices. This is theoretically linked to better generalization bounds, meaning the model is less likely to overfit and performs better on unseen data.
  • Overlapping Clusters are Realistic: Allowing items to belong to multiple groups better reflects the nuanced nature of item relationships, leading to more robust representations.
  • Cold-Start Advantage: The structure learning approach provides a significant boost in cold-start performance, suggesting that the learned item relationships are robust even when user data is limited.

Key Results

  • State-of-the-Art Performance: SW-DAE consistently outperformed state-of-the-art baselines like MULT-VAE and MULT-DAE across all three datasets in terms of NDCG@100, Recall@20, and Recall@50.
  • Comparable Resources, Superior Results: Even when SW-DAE-P (a variant with the same number of parameters and FLOPs as MULT-VAE) was used, it still achieved considerable performance gains.
  • Improved Cold-Start: SW-DAE showed a notable improvement in cold-start scenarios compared to MULT-VAE/DAE.
  • Scalability: The item grouping algorithms (Algorithm 1 and 2) were shown to be fast and scalable, even on large and sparse datasets.
  • Qualitative Validation: Analysis of the learned neuron clusters revealed that they indeed captured thematically similar items and genres, validating the effectiveness of the item grouping strategy.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper mentions searching for hyperparameters, the sensitivity of the proposed method to parameters like K (number of clusters) and R (degree of overlap) is an area for further exploration. Figure 7 suggests an optimal sparsity range, indicating that extreme sparsity or density can be detrimental.
  • Greedy Layer-wise Training for Deeper Networks: The paper outlines a greedy layer-wise approach for building deeper SW-DAEs. While this is a promising direction, the performance and effectiveness of deeper, stacked SW-DAEs would benefit from more extensive empirical validation.
  • Exploration of Clustering Methods: The paper mentions that other distance-based clustering methods could be explored beyond K-means for item grouping.
  • Generalization to Other Domains: While successful in recommendation, the applicability of this structure learning approach to other domains that can be framed as autoencoding tasks could be investigated.

Summary updated

Abstract

Autoencoder recommenders have recently shown state-of-the-art performance in the recommendation task due to their ability to model non-linear item relationships effectively. However, existing autoencoder recommenders use fully-connected neural network layers and do not employ structure learning. This can lead to inefficient training, especially when the data is sparse as commonly found in collaborative filtering. The aforementioned results in lower generalization ability and reduced performance. In this paper, we introduce structure learning for autoencoder recommenders by taking advantage of the inherent item groups present in the collaborative filtering domain. Due to the nature of items in general, we know that certain items are more related to each other than to other items. Based on this, we propose a method that first learns groups of related items and then uses this information to determine the connectivity structure of an auto-encoding neural network. This results in a network that is sparsely connected. This sparse structure can be viewed as a prior that guides the network training. Empirically we demonstrate that the proposed structure learning enables the autoencoder to converge to a local optimum with a much smaller spectral norm and generalization error bound than the fully-connected network. The resultant sparse network considerably outperforms the state-of-the-art methods like Mult-vae/Mult-dae on multiple benchmarked datasets even when the same number of parameters and flops are used. It also has a better cold-start performance.
BibTeX
@inproceedings{khawar2020learning,
  author = {Farhan Khawar and Leonard K. M. Poon and Nevin L. Zhang},
  title = {{Learning the Structure of Auto-Encoding Recommenders}},
  booktitle = {The Web Conference},
  year = {2020},
  url = {https://openreview.net/forum?id=0ijJcxzKBqy},
}
Figure