Abstract
Topic modelling is a popular unsupervised method for identifying the underlying themes in document collections that has many applications in information retrieval. A topic is usually represented by a list of terms ranked by their probability but, since these can be difficult to interpret, various approaches have been developed to assign descriptive labels to topics. Previous work on the automatic assignment of labels to topics has relied on a two-stage approach: (1) candidate labels are retrieved from a large pool (e.g. Wikipedia article titles); and then (2) re-ranked based on their semantic similarity to the topic terms. However, these extractive approaches can only assign candidate labels from a restricted set that may not include any suitable ones. This paper proposes using a sequence-to-sequence neural-based approach to generate labels that does not suffer from this limitation. The model is trained over a new large synthetic dataset created using distant supervision. The method is evaluated by comparing the labels it generates to ones rated by humans.
Introduction
Probabilistic topic models, such as Latent Dirichlet Allocation (LDA), are powerful tools for uncovering latent themes in massive text corpora. However, the standard output—a list of terms ranked by probability—is often difficult for non-experts to interpret. For instance, a list like {pain, disorder, symptom, depression, anxiety} clearly refers to "Mental Health," but identifying this requires significant cognitive effort.
Traditional automatic labeling methods are primarily extractive. They rely on a "retrieve and rank" pipeline where candidate labels are pulled from a reference set (like Wikipedia titles) and then sorted by relevance. The fundamental limitation of this approach is its fixed vocabulary: if the ideal label is not present in the reference set, the model cannot produce it. This research introduces a generative approach, using neural networks to synthesize descriptive labels from scratch, effectively treating topic labeling as a translation task.
Methodology: A Generative Approach
The authors propose a sequence-to-sequence (seq2seq) architecture to translate a set of topic terms into a concise label. This model moves beyond the constraints of a fixed candidate pool by generating labels word-by-word.
The architecture consists of three main components:
- Encoder: A bidirectional Gated Recurrent Unit (Bi-GRU) that processes the top $N$ terms of a topic to create a rich representation of the input.
- Attention Mechanism: This component allows the decoder to dynamically focus on specific input terms while generating different parts of the label, ensuring that the most relevant terms (e.g., "depression") strongly influence the resulting label.
- Decoder: A GRU that predicts the label sequence word-by-word, conditioned on the context vector provided by the encoder and attention layer.
Data and Distant Supervision
Training neural networks requires substantial amounts of labeled data, which is scarce for topic modeling. To overcome this, the authors employed distant supervision to create large-scale synthetic datasets from Wikipedia.
Two primary synthetic datasets were developed:
- ds_wiki_tfidf: Wikipedia article titles serve as labels, with the top 30 words (ranked by TF-IDF) acting as topic terms.
- ds_wiki_sent: Similar to the above, but the topic terms are derived from the first 30 words of the article.
The models were evaluated against topics_bhatia, a gold-standard dataset containing 228 topics with human-rated labels.
Key Results and Insights
The performance of the generative model was measured using BERTScore, which captures semantic similarity rather than simple lexical overlap.
- Superiority of Generative Models: The seq2seq approach consistently outperformed extractive baselines. The best configuration, trained on the
ds_wiki_sentdataset, achieved a BERTScore F-measure of 0.929, significantly higher than the strongest baseline (0.902). - Contextual Advantage: The
ds_wiki_sentdataset proved more effective than the TF-IDF version. This suggests that the narrative structure of an introductory paragraph provides a better signal for topic labeling than a simple bag-of-words approach. - Input Sensitivity: Increasing the number of input terms from 10 to 30 led to a marked improvement in label quality, providing the model with more context to synthesize accurate descriptions.
Limitations and Future Work
While the generative approach is a significant step forward, it faces challenges. The model is sensitive to the training domain; since it was trained on Wikipedia, it occasionally struggles with highly technical or niche domains. Furthermore, the quality of the label is inherently tied to the quality of the underlying topic; "junk" topics consisting of unrelated words naturally lead to nonsensical labels. Future research aims to improve label appropriateness by incorporating domain-specific data and exploring more advanced transformer-based architectures.
Summary updated