Semantic Language Models for Topic Detection and Tracking

Ramesh NallapatiNAACL 2003 · OpenReview

Abstract: In this work, we present a new semantic language modeling approach to model news stories in the Topic Detection and Tracking (TDT) task. In the new approach, we build a unigram language model for each semantic class in a news story. We also cast the link detection subtask of TDT as a two-class classification problem in which the features of each sample consist of the generative log-likelihood ratios from each semantic class. We then compute a linear discriminant classifier using the perceptron learning algorithm on the training set. Results on the test set show a marginal improvement over the unigram performance, but are not very encouraging on the whole.

Introduction

Topic Detection and Tracking (TDT) aims to automatically organize news streams into cohesive events. A fundamental challenge in this domain is "link detection"—determining whether two distinct news stories cover the same underlying topic. Traditional approaches have largely relied on the Vector Space Model (using cosine similarity) or standard Unigram Language Models.

However, these models typically treat all words as having equal semantic weight. In the context of news, specific entities (e.g., a person's name) or actions (e.g., a specific verb) are often more indicative of a topic than generic adjectives or adverbs. This research investigates a principled framework for integrating high-level NLP features, such as Part-of-Speech (POS) tags and Named Entities (NE), directly into a probabilistic language modeling framework to improve link detection performance.

Methodology

The proposed Semantic Language Model operates in two distinct stages:

  1. Class-Specific Likelihood Ratios: Rather than building a single model for an entire story, the framework defines specific semantic classes (e.g., Persons, Locations, Nouns, Verbs). For any given story pair ($D_1, D_2$), a log-likelihood ratio is calculated for each class. This ratio quantifies how much more likely the words in $D_2$ were generated by the topic model of $D_1$ relative to a general English background model.
  2. Weighted Perceptron Integration: To synthesize these disparate semantic signals, link detection is treated as a binary classification problem. A linear perceptron is employed to learn the optimal weights for each class's likelihood ratio, effectively determining which semantic categories are most discriminative for identifying related topics.
Architecture of the Semantic Language Model

To mitigate errors from automatic tagging (such as the misidentification of a Person as an Organization), the model also incorporates "unions" of classes, such as a combined "Person + Organization + Location" category, to provide more robust features.

Experimental Setup

The model was evaluated using standard TDT benchmarks:

  • Training Set: A subset of the TDT3 corpus, comprising 7,200 story pairs from eight English news sources (October–December 1998).
  • Test Set: A randomly selected subset of the TDT2 corpus, consisting of 6,363 story pairs.
  • Background Models: General English statistics were derived from large-scale manually transcribed news collections (37,526 stories for TDT3 and 40,851 for TDT2).

Key Insights and Analysis

The analysis of the learned perceptron weights revealed several critical findings regarding the utility of semantic information in TDT:

  • Dominance of the "All" Class: The perceptron assigned the highest weight to the "All" category, which represents the standard unigram model. This suggests that the raw distribution of all words remains the most robust signal for topic matching, likely because standard unigram models already implicitly capture semantic importance through mechanisms similar to tf-idf weighting.
  • Discriminative Power of Nouns and Verbs: Interestingly, the combined class of Nouns, Verbs, and Adjectives received higher weights than Named Entity classes (Persons, Organizations, Locations). This implies that the actions and objects described in a story are often more discriminative for link detection than the specific entities involved.
  • Tagging Robustness: The use of class unions proved beneficial in offsetting the noise introduced by automatic NLP tagging tools.
Learned Perceptron weights for semantic classes

Results and Conclusion

The quantitative results were modest. On the training set, the semantic model achieved a Minimum Cost of 0.1200, slightly trailing the baseline unigram's 0.1136. On the test set, the semantic model showed a marginal improvement, with a minimum cost of 0.1248 compared to the unigram's 0.1260.

While the framework is theoretically sound, the added complexity of semantic modeling did not yield the expected performance leap. This may be due to the limitations of a linear classifier or the use of uniform smoothing across all semantic classes. Future work will explore non-linear classifiers like Support Vector Machines (SVMs) and class-specific smoothing parameters to better leverage semantic information in TDT tasks.

Summary updated

Abstract

In this work, we present a new semantic language modeling approach to model news stories in the Topic Detection and Tracking (TDT) task. In the new approach, we build a unigram language model for each semantic class in a news story. We also cast the link detection subtask of TDT as a two-class classification problem in which the features of each sample consist of the generative log-likelihood ratios from each semantic class. We then compute a linear discriminant classifier using the perceptron learning algorithm on the training set. Results on the test set show a marginal improvement over the unigram performance, but are not very encouraging on the whole.
BibTeX
@inproceedings{nallapati2003semantic,
  author = {Ramesh Nallapati},
  title = {{Semantic Language Models for Topic Detection and Tracking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
  url = {https://openreview.net/forum?id=S1-DE7-ubH},
}
Figure