ARTM vs. LDA: an SVD Extension Case Study

Sergey I. NikolenkoAIST (Supplement) 2016 · OpenReview

Abstract

This study compares two frameworks for extending topic models in recommender systems: Latent Dirichlet Allocation (LDA) and Additive Regularization of Topic Models (ARTM). While integrating Singular Value Decomposition (SVD) with LDA necessitates a complex first-order Taylor approximation for Gibbs sampling, the ARTM framework achieves the same result through straightforward differentiation of a regularizer. We demonstrate that ARTM offers a more mathematically natural and extensible approach for building hybrid topic models.

Introduction

Topic models like LDA are powerful tools for unsupervised text analysis but often require augmentation with external data for specific applications, such as article recommendation. A common approach is to combine topic modeling with collaborative filtering, specifically Singular Value Decomposition (SVD), to create a joint model (SVD-LDA) that leverages both content and user preferences.

However, performing inference on such hybrid models is computationally challenging. The use of a logistic sigmoid function for binary rating prediction makes standard Gibbs sampling intractable. Previous solutions relied on a complex first-order Taylor approximation to maintain feasibility. This paper investigates whether the Additive Regularization of Topic Models (ARTM) framework provides a more elegant mathematical foundation for these extensions.

Plate diagrams for (a) LDA and (b) the joint SVD-LDA model.

Methodology

The authors compare the inference derivations between two primary frameworks:

  1. SVD-LDA (Baseline): This model uses collapsed Gibbs sampling. To handle the intractable logistic sigmoid, a first-order Taylor series approximation at zero is employed. This allows for the precomputation of topic vectors per document, making sampling feasible but mathematically cumbersome.
  2. SVD-ARTM (Proposed Approach): ARTM extends probabilistic Latent Semantic Analysis (pLSA) by incorporating regularizers into the objective function. It utilizes an EM-like iterative algorithm (Newton's method) for optimization. The authors define a new regularizer based on the SVD recommendation likelihood and compute its partial derivatives with respect to topic parameters.

The central finding is that the derivative of the ARTM regularizer is mathematically identical to the Taylor approximation used in the LDA-based Gibbs sampler.

Key Insights and Results

The theoretical parity between the two approaches is the paper's primary contribution. The complex approximation required for SVD-LDA naturally emerges from the ARTM framework's objective function. This suggests that ARTM is a superior framework for rapid prototyping, as adding constraints like collaborative filtering only requires defining a regularizer and its derivative, avoiding the complexities of custom MCMC design.

Quantitative results from previous work justify the hybrid approach. As shown in the table below, the joint model significantly outperforms pure SVD across multiple metrics.

Performance comparison of pure SVD, SVD-LDA, and SVD-LDA-DEM models.

The hybrid SVD-LDA-DEM model (50 topics, 10 features) achieved an NDCG of 0.9840 and an AUC of 0.8901, outperforming the baseline SVD. Since the ARTM derivation leads to the same update rules, it is expected to achieve identical performance with a more streamlined implementation.

Conclusion

This case study demonstrates that ARTM provides a mathematically elegant alternative to LDA for complex topic model extensions. By bypassing the need for intricate sampling approximations, ARTM facilitates the development of sophisticated hybrid models. Future work will focus on implementing the SVD regularizer within the open-source BigARTM library to enable large-scale empirical validation.

Summary updated

BibTeX
@inproceedings{nikolenko2016artm,
  author = {Sergey I. Nikolenko},
  title = {{ARTM vs. LDA: an SVD Extension Case Study}},
  booktitle = {AIST (Supplement)},
  year = {2016},
  url = {https://openreview.net/forum?id=xli5D9WKbhx},
}
Figure