Abstract
We present a new topic model that generates documents by sampling a topic for one whole sentence at a time, and generating the words in the sentence using an RNN decoder that is conditioned on the topic of the sentence. We argue that this novel formalism will help us not only visualize and model the topical discourse structure in a document better, but also potentially lead to more interpretable topics since we can now illustrate topics by sampling representative sentences instead of bag of words or phrases. We present a variational auto-encoder approach for learning in which we use a factorized variational encoder that independently models the posterior over topical mixture vectors of documents using a feed-forward network, and the posterior over topic assignments to sentences using an RNN. Our preliminary experiments on two different datasets indicate early promise, but also expose many challenges that remain to be addressed.
Introduction
Traditional topic modeling, exemplified by Latent Dirichlet Allocation (LDA), has long relied on the "bag-of-words" (BoW) assumption. By treating documents as unordered collections of tokens, these models ignore the inherent syntactic structure and logical progression of natural language. While BoW models are effective for broad categorization, they struggle to capture the nuances of how themes evolve within a text or how specific word choices are constrained by local context.
Recent advancements in neural topic modeling, such as Neural Variational Document Models (NVDM), have integrated Variational Auto-Encoders (VAEs) to improve performance. However, many of these models still adhere to the BoW formalism. The SenGen model proposes a shift in perspective: treating the sentence as the fundamental unit of topic modeling. Sentences are typically topically cohesive, and shifts in subject matter generally occur at sentence or paragraph boundaries. By modeling topics at the sentence level, we can better capture discourse structure and improve topic interpretability.
The SenGen Model
SenGen is a neural variational topic model designed to generate documents sentence-by-sentence. This approach allows the model to explicitly represent the topical flow of a document.
Generative Process
The generative process in SenGen follows a hierarchical structure:
- Document-level Topic Strengths: For each document $d$, a topic mixture vector $\theta_d$ is sampled from a Gaussian distribution.
- Sentence-level Topic Indicators: For each sentence $s$ in the document, a specific topic index $z_s$ is sampled based on the document's topic strengths $\theta_d$.
- Word Generation: The words of the sentence are generated using a Gated Recurrent Unit (GRU) decoder. Crucially, this decoder is conditioned on the chosen sentence topic $z_s$ via a learned topic-embedding vector.
Variational Inference and Architecture
To facilitate learning, SenGen employs a factorized encoder architecture within a VAE framework. The encoder is split into two components:
- Document Encoder: A feed-forward network that estimates the posterior distribution of the document-level topic mixture.
- Sentence Encoder: A GRU-RNN that models the posterior probability of a topic assignment given the sequence of words in a sentence.
The decoder uses topic-specific softmax layers. To maintain efficiency, most parameters are shared across topics, but the final output layer is unique to each topic, allowing for distinct vocabularies and stylistic "voices" for different themes.
Experimental Evaluation
The authors evaluated SenGen on two datasets where sentence boundaries were preserved: the 20 Newsgroups corpus and a subsampled version of the CNN/Daily Mail news archive.
Quantitative Evaluation
The performance of SenGen was measured using perplexity, a standard metric for language models. The results, compared against traditional LDA and other neural variants, were mixed.
On the CNN/Daily Mail dataset, SenGen (671) outperformed both LDA (776) and ProdLDA (735), demonstrating the benefit of modeling sentence structure in large-scale news text. However, it trailed behind NVDM (435). On the 20 Newsgroups dataset, SenGen struggled significantly, likely due to the high noise level (e.g., email headers and signatures) and the model's tendency to overfit on smaller, noisier corpora.
Qualitative Observations
A key motivation for SenGen is interpretability. Instead of representing topics as lists of keywords, SenGen allows for the generation of representative sentences. While beam search often resulted in generic, repetitive phrases, stochastic sampling revealed that the model successfully learned to associate topical vocabulary with specific sentence-level indicators. This suggests that while the model captures topical essence, generating perfectly grammatical and diverse sentences remains a challenge.
Challenges and Future Work
The development of SenGen highlighted several significant hurdles in sentence-level topic modeling:
- Computational Complexity: The use of RNN decoders with large vocabularies leads to a massive parameter space (up to 159 million parameters), making training computationally expensive.
- Topic Coherence: Learned topics currently lack the coherence of simpler BoW models, often due to the model's sensitivity to frequent stop words.
- Independence Assumptions: The current iteration assumes sentence topics are independent. Future iterations will explore modeling dependencies between adjacent sentences to capture more natural thematic transitions.
In conclusion, SenGen represents a significant step toward more structurally aware topic models. By moving beyond the bag-of-words, it provides a framework for understanding documents as coherent sequences of ideas, paving the way for more interpretable and contextually rich text analysis.
Summary updated