Automatic Generation of Topic Labels

Areej Alokaili, Nikolaos Aletras, Mark Stevenson

Abstract

Topic modelling is a popular unsupervised method for identifying the underlying themes in document collections that has many applications in information retrieval. A topic is usually represented by a list of terms ranked by their probability but, since these can be difficult to interpret, various approaches have been developed to assign descriptive labels to topics. Previous work on the automatic assignment of labels to topics has relied on a two-stage approach: (1) candidate labels are retrieved from a large pool (e.g. Wikipedia article titles); and then (2) re-ranked based on their semantic similarity to the topic terms. However, these extractive approaches can only assign candidate labels from a restricted set that may not include any suitable ones. This paper proposes using a sequence-to-sequence neural-based approach to generate labels that does not suffer from this limitation. The model is trained over a new large synthetic dataset created using distant supervision. The method is evaluated by comparing the labels it generates to ones rated by humans.

Introduction

Probabilistic topic models, such as Latent Dirichlet Allocation (LDA), are powerful tools for discovering latent thematic structures in large text corpora. However, the standard output—a list of keywords ranked by probability—often requires significant cognitive effort for human interpretation. To bridge this gap, automatic topic labeling aims to provide concise, human-readable descriptions.

Historically, this has been approached as an extractive task. Systems search a reference database, such as Wikipedia, for candidate titles and rank them based on their semantic proximity to the topic terms. The primary drawback of this paradigm is its dependency on the candidate pool; if a precise label does not exist in the database, the system cannot produce it. This research shifts the focus to a generative approach, treating topic labeling as a sequence-to-sequence (seq2seq) translation task where keywords are "translated" into a descriptive label.

Methodology

The proposed model utilizes a seq2seq architecture enhanced with an attention mechanism, which allows the system to synthesize labels from scratch rather than selecting them from a fixed list.

The seq2seq architecture with attention used for topic label generation.

The architecture consists of three primary components:

  1. BiGRU Encoder: A bidirectional Gated Recurrent Unit (GRU) processes the input sequence of topic terms. By reading the sequence in both directions, the encoder captures contextual dependencies between keywords regardless of their rank.
  2. Attention Mechanism: This layer allows the decoder to dynamically weigh the importance of different input terms when generating each word of the output label. It ensures that highly descriptive terms have a greater influence on the final label than more general terms.
  3. GRU Decoder: The decoder generates the label word-by-word, conditioned on the context vector from the attention layer and the previously generated tokens.

Data and Distant Supervision

Training neural networks for this task is challenging due to the lack of large-scale, human-labeled topic datasets. To overcome this, the authors employed distant supervision to generate synthetic training data from approximately 300,000 Wikipedia articles. Two distinct datasets were created:

  • ds_wiki_tfidf: Topic terms were derived from the top 30 words of an article ranked by TF-IDF, with the article title serving as the target label.
  • ds_wiki_sent: Topic terms were taken from the first 30 words of the article, providing a more narrative, context-rich input.

Evaluation was conducted using the topics_bhatia dataset, which includes 228 real-world topics from diverse sources like news and medical journals, paired with human-validated gold labels.

Results and Discussion

The performance of the generative models was evaluated using BERTScore, which measures the semantic similarity between generated labels and gold-standard labels using contextual embeddings.

BERTScore results comparing the proposed models against baselines.

As shown in the results table, the seq2seq models consistently outperformed extractive baselines. Key findings include:

  • Contextual Superiority: Models trained on ds_wiki_sent generally performed better than those trained on ds_wiki_tfidf. This indicates that the natural linguistic context found in article introductions is more informative for label generation than raw statistical word frequencies.
  • Input Density: Increasing the number of input topic terms from 10 to 30 improved performance, as the additional terms provided a richer semantic signal for the encoder.
  • Generative Flexibility: Qualitative analysis showed the model's ability to synthesize novel labels. For instance, it generated "military force" for a cluster of war-related terms, demonstrating a capacity for abstraction beyond simple keyword selection.

Conclusion

This work demonstrates that generative seq2seq models are a viable and effective alternative to traditional extractive topic labeling. By leveraging distant supervision on large-scale data, the model learns to produce descriptive, human-like labels.

However, limitations remain. The model's performance is sensitive to the coherence of the underlying topic; noisy or unrelated word clusters often result in generic or nonsensical labels. Furthermore, domain-specific topics can be challenging if they are underrepresented in the Wikipedia-based training data. Future research will focus on improving synthetic data quality and enhancing the model's robustness across specialized domains.

Summary updated

Abstract

Topic modelling is a popular unsupervised method for identifying the underlying themes in document collections that has many applications in information retrieval. A topic is usually represented by a list of terms ranked by their probability but, since these can be difficult to interpret, various approaches have been developed to assign descriptive labels to topics. Previous work on the automatic assignment of labels to topics has relied on a two-stage approach: (1) candidate labels are retrieved from a large pool (e.g. Wikipedia article titles); and then (2) re-ranked based on their semantic similarity to the topic terms. However, these extractive approaches can only assign candidate labels from a restricted set that may not include any suitable ones. This paper proposes using a sequence-to-sequence neural-based approach to generate labels that does not suffer from this limitation. The model is trained over a new large synthetic dataset created using distant supervision. The method is evaluated by comparing the labels it generates to ones rated by humans.
BibTeX
@inproceedings{alokaili2020automatic,
  author = {Areej Alokaili and Nikolaos Aletras and Mark Stevenson},
  title = {{Automatic Generation of Topic Labels}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  url = {https://openreview.net/forum?id=5xDIlqHU9uf},
}
Figure