CompareLDA: A Topic Model for Document Comparison

Maksim Tkachenko, Hady W. LauwAAAI 2019 · OpenReview

Abstract

A number of real-world applications require comparison of entities based on their textual representations. In this work, we develop a topic model supervised by pairwise comparisons of documents. Such a model seeks to yield topics that help to differentiate entities along some dimension of interest, which may vary from one application to another. While previous supervised topic models consider document labels in an independent and pointwise manner, our proposed Comparative Latent Dirichlet Allocation (CompareLDA) learns predictive topic distributions that comply with the pairwise comparison observations. To fit the model, we derive a maximum likelihood estimation method via augmented variational approximation algorithm. Evaluation on several public datasets underscores the strengths of CompareLDA in modelling document comparisons.

Introduction

Traditional topic models, such as Latent Dirichlet Allocation (LDA), are typically unsupervised, identifying themes based on word co-occurrence patterns. While effective for general summarization, these themes often fail to align with specific analytical goals. For example, in analyzing country reports for healthcare outcomes, standard LDA might prioritize geographical or linguistic topics over medical infrastructure.

Supervised variants like sLDA attempt to address this by incorporating pointwise labels (e.g., numerical ratings). However, absolute ratings are frequently noisy, subjective, or unavailable. In many practical scenarios—such as choosing between grant proposals or selecting one product over another—only relative preference information is accessible. CompareLDA addresses this by utilizing pairwise comparisons to guide the discovery of "discriminative" topics that explain the differences between entities.

Methodology

CompareLDA extends the LDA framework by introducing a "merit" component for each document. The generative process involves four primary stages:

  1. Topic Discovery: Documents are modeled as mixtures of latent topics.
  2. Merit Mapping: An empirical topic distribution is calculated for each document and mapped to a scalar merit value ($m$) via linear regression with learned weights ($\eta$).
  3. Pairwise Supervision: The probability of one document "winning" a comparison is modeled using a sigmoid function of the difference between their respective merit values.
  4. Joint Learning: Unlike pipeline approaches, CompareLDA optimizes topic discovery and merit ranking simultaneously.
Graphical model of CompareLDA showing the interaction between topics, words, and comparison labels.

To estimate parameters, the authors developed a maximum likelihood method using an augmented variational approximation. This is combined with a Metropolis-Hastings sampling procedure (simulated annealing) to manage the non-linear interactions between document pairs.

Experimental Setup

The model was evaluated on five datasets from three sources:

  • Wikipedia: Countries ranked by Alcohol Consumption, Cigarette Consumption, and Life Expectancy.
  • Amazon Electronics: Comparisons between "positive" (4-5 stars) and "negative" (1-2 stars) reviews.
  • Movie Reviews: Comparisons based on star ratings.

To simulate realistic conditions, the authors often used only a small fraction (e.g., 0.25%) of possible pairwise comparisons for training.

Results and Discussion

CompareLDA consistently outperformed sLDA in both ranking accuracy and topic quality. For instance, on the Wikipedia Alcohol Consumption dataset, CompareLDA achieved approximately 78.8% accuracy, while sLDA performed near random at 55%.

Performance comparison between CompareLDA and sLDA across varying numbers of topics, showing (a) Accuracy and (b) Per-word Held-out Log-likelihood.

Key findings include:

  • Relative Supervision Efficiency: Treating absolute ratings as pairwise comparisons often yields better results than pointwise modeling. Accuracy stabilizes even with only 5% of possible comparisons.
  • Semantic Discriminability: The model successfully isolates "good" and "bad" qualities. In product reviews, positive topics included terms like "great" and "easy," while negative topics highlighted "warranty" and "problem."
  • Model Robustness: CompareLDA showed lower variance and higher stability as the number of topics increased compared to baselines. The higher held-out log-likelihood suggests that pairwise supervision helps discover more coherent topics rather than simply overfitting.

Conclusion

CompareLDA demonstrates the power of relative supervision in topic modeling. While the Metropolis-Hastings sampling adds computational complexity compared to standard variational inference, the gains in discriminative power and topic coherence are significant. Future work may explore non-linear merit mappings, such as neural networks, and methods to improve performance for documents with sparse text.

Summary updated

Abstract

A number of real-world applications require comparison of entities based on their textual representations. In this work, we develop a topic model supervised by pairwise comparisons of documents. Such a model seeks to yield topics that help to differentiate entities along some dimension of interest, which may vary from one application to another. While previous supervised topic models consider document labels in an independent and pointwise manner, our proposed Comparative Latent Dirichlet Allocation (CompareLDA) learns predictive topic distributions that comply with the pairwise comparison observations. To fit the model, we derive a maximum likelihood estimation method via augmented variational approximation algorithm. Evaluation on several public datasets underscores the strengths of CompareLDA in modelling document comparisons.
BibTeX
@inproceedings{tkachenko2019comparelda,
  author = {Maksim Tkachenko and Hady W. Lauw},
  title = {{CompareLDA: A Topic Model for Document Comparison}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=0W67Xilwtc},
}
Figure