Replicated Softmax: an Undirected Topic Model

Ruslan Salakhutdinov, Geoffrey E. HintonNeurIPS 2009 · OpenReview

Topic modeling has long been dominated by directed graphical models like Latent Dirichlet Allocation (LDA). While popular, these models suffer from computational bottlenecks and representational limitations. The paper Replicated Softmax: an Undirected Topic Model introduces a powerful alternative: a two-layer, undirected graphical model based on the Restricted Boltzmann Machine (RBM) architecture. By employing a clever weight-sharing mechanism, this model achieves better generalization, faster retrieval, and naturally handles documents of varying lengths.

The Problem with Directed Topic Models

Classic directed topic models like LDA face two primary challenges:

  1. Intractable Inference: Calculating the exact topic distribution of a document is computationally prohibitive. Researchers are forced to rely on slow or inaccurate approximations (like variational inference or Gibbs sampling).
  2. The "Blurry" Mixture Problem: Directed models treat documents as a mixture of topics, meaning they can never predict a word more precisely than their sharpest individual topic. They struggle to learn "distributed representations"—the concept that multiple broad topics (e.g., "government" and "mafia") should combine to point specifically to a single, highly precise word (e.g., "Berlusconi").

Previous attempts to use undirected models, such as standard RBMs, failed in natural language processing because they could not handle documents of different lengths. A model trained on short emails would break down when processing a long research paper.

The Architecture of Replicated Softmax

To solve these issues, the authors propose the Replicated Softmax model. It is a two-layer undirected model consisting of:

  • Visible Units: Representing the words in a document.
  • Hidden Units: Representing the latent "topics."
The Replicated Softmax architecture alongside standard and multinomial visible models.

The breakthrough lies in the "replicated" nature of the visible layer. For a document of length $D$, the model essentially creates $D$ separate softmax units that all share the same weights connecting them to the hidden units. Mathematically, this is equivalent to having a single multinomial unit sampled $D$ times.

Crucially, to maintain stability across varying document sizes, the authors introduce a scaling trick: they scale the biases of the hidden units by the document length $D$. This ensures that the hidden topic units behave consistently whether the document contains ten words or ten thousand.

For training, the model utilizes Contrastive Divergence (CD), an efficient approximation for gradient descent in RBMs. To evaluate the model—which requires calculating the difficult partition function—they employ Annealed Importance Sampling (AIS).

Key Insights

  • Distributed Power: Unlike LDA, where topics compete to explain a word, Replicated Softmax allows topics to cooperate. Multiple active hidden features multiply their distributions together, allowing the model to pinpoint specific words much more precisely than any single topic could alone.
  • Length Invariance: By scaling hidden biases by document length, the authors solved the primary stability issue that previously prevented undirected models from being viable in NLP.
  • Computational Efficiency: Despite the complex underlying math, inference is remarkably efficient. Computing the topic features for a document requires just a single matrix multiplication followed by a sigmoid function, bypassing the slow, iterative inference required by LDA.

Key Results and Performance

The model was evaluated on three standard text benchmarks: NIPS Proceedings, 20-Newsgroups, and the Reuters Corpus Volume I (RCV1-v2). The authors compared a 50-topic Replicated Softmax against LDA models with 50 and 200 topics, measuring perplexity (how well the model predicts held-out data, where lower is better).

  • Better Generalization: On the massive Reuters dataset, Replicated Softmax achieved a perplexity of 988, significantly outperforming both LDA-50 (1437) and LDA-200 (1142).
  • Superior Retrieval: In document retrieval tasks (finding similar documents in a database given a query), the Replicated Softmax consistently demonstrated higher precision at various recall levels compared to LDA.
Scatter plots comparing test perplexity per document for LDA versus Replicated Softmax across three datasets.

Furthermore, scatter plots of test perplexity per document revealed that Replicated Softmax outperformed LDA on almost every single individual document, proving its consistency across different types of text.

Limitations and Future Directions

The paper presents the simplest possible version of this model—a shallow, two-layer architecture. While the authors suggest that stochastic gradient descent could scale this approach to billions of documents, the experiments were conducted on standard, more modest scales.

Looking forward, the undirected nature of the Replicated Softmax makes it an ideal building block for Deep Belief Networks (DBNs). By stacking more layers, future models could extract even more abstract semantic representations. Additionally, the framework naturally supports "supervised" extensions, allowing document labels or metadata (like authors and dates) to be incorporated directly into the topic modeling process.

Summary updated

Abstract

We show how to model documents as bags of words using family of two-layer, undirected graphical models. Each member of the family has the same number of binary hidden units but a different number of ``softmax visible units. All of the softmax units in all of the models in the family share the same weights to the binary hidden units. We describe efficient inference and learning procedures for such a family. Each member of the family models the probability distribution of documents of a specific length as a product of topic-specific distributions rather than as a mixture and this gives much better generalization than Latent Dirichlet Allocation for modeling the log probabilities of held-out documents. The low-dimensional topic vectors learned by the undirected family are also much better than LDA topic vectors for retrieving documents that are similar to a query document. The learned topics are more general than those found by LDA because precision is achieved by intersecting many general topics rather than by selecting a single precise topic to generate each word.
BibTeX
@inproceedings{salakhutdinov2009replicated,
  author = {Ruslan Salakhutdinov and Geoffrey E. Hinton},
  title = {{Replicated Softmax: an Undirected Topic Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=ry-HpP-_-H},
}
Figure