A Latent Topic Extracting Method based on Events in a Document and its Application

Risa Kitajima, Ichiro KobayashiACL 2011

Sure, here's a blog-style summary of the paper "A Latent Topic Extracting Method based on Events in a Document and its Application":

TL;DR

This paper introduces "eventLDA," a novel latent topic modeling approach that treats documents as collections of "events" (pairs of related words) rather than just individual words. This event-centric view allows for a more nuanced understanding of document content, leading to improved performance in tasks like document retrieval and summarization compared to traditional word-based methods.

What Problem Does This Paper Address?

Traditional latent topic analysis methods, such as Latent Semantic Indexing (LSI), Probabilistic LSI (pLSI), and Latent Dirichlet Allocation (LDA), are widely used for text analysis. However, these methods primarily focus on assigning topics to individual words within a document. This approach can overlook the crucial relationships and dependencies between words that often convey the true meaning or "event" of a sentence. For instance, two documents might have similar word frequencies but express entirely different sentiments or ideas due to the way words are combined. The paper aims to address this limitation by proposing a method that captures topics at the level of "events" within a document, leading to a more accurate representation of its content.

Methods

The core innovation of this paper is the introduction of "eventLDA." Instead of treating documents as bags of words, it defines an "Event" as a pair of words in a dependency relationship, specifically (Subject, Predicate) or (Predicate1, Predicate2). These events are extracted using a Japanese dependency structure analyzer.

The process involves:

  1. Event Extraction: Identifying dependency pairs within sentences.
  2. Event-by-Document Matrix Creation: Constructing a matrix where rows represent unique events and columns represent documents, with cell values indicating the frequency of an event in a document. Unlike traditional word-based methods, all extracted events are included, even those appearing only once, to ensure the document vector accurately reflects its content.
  3. Latent Topic Estimation: Applying Latent Dirichlet Allocation (LDA) to this event-by-document matrix. In this adaptation, each topic is represented as a multinomial distribution of events, rather than words. The authors use Gibbs sampling for topic distribution estimation.

For evaluation, they compare "eventLDA" against a conventional word-based LDA ("wordLDA") using document retrieval and summarization tasks. Similarity measures like Kullback-Leibler Divergence, Symmetric Kullback-Leibler Divergence, Jensen-Shannon Divergence, and cosine similarity are employed.

Datasets

  • Document Retrieval Experiment: A dataset of user reviews and facility evaluations for hotels from Rakuten Travel was used. This dataset consists of 2000 documents, with 1000 randomly selected from reviews rating "a room" as 1 (bad) and 1000 from reviews rating it as 5 (good). The query used was "a room is good."
  • Summarization Experiment: The NTCIR4 (NII Test Collection for IR Systems 4) TSC3 (Text Summarization Challenge 3) dataset was utilized. This dataset comprises 30 topic sets, each containing approximately 10 Japanese newspaper articles, totaling 3587 sentences. Queries were derived from existing evaluation questions for summarization results.

Key Insights and Ideas

  • Events as Meaningful Units: The paper posits that word pairs in dependency relationships (Events) are more meaningful units for capturing document content than individual words, especially when dealing with nuanced topics.
  • Improved Topic Representation: By focusing on events, eventLDA can provide a more detailed classification of documents with a smaller number of topics compared to wordLDA. This is attributed to the reduced ambiguity when topics are defined over events rather than individual words.
  • Cosine Similarity as Optimal Measure: Surprisingly, cosine similarity, which is not typically used for estimating probabilistic distributions, yielded the best performance for eventLDA in the document retrieval task. This suggests that the nature of event-based topic distributions might differ from word-based ones.
  • EventLDA's Robustness: The accuracy of eventLDA in the summarization task was less dependent on the specific similarity measure used, indicating a more stable and robust topic estimation.

Key Results

  • Document Retrieval: eventLDA consistently outperformed wordLDA across various numbers of topics and similarity measures in terms of 11-point interpolated average precision. For instance, with an optimal number of topics (k=5 for eventLDA, k=50 for wordLDA), eventLDA achieved higher accuracy.
  • Summarization: In multi-document summarization, eventLDA achieved comparable or superior performance to traditional methods like the Lead method and TF-IDF based summarization, particularly in terms of coverage. eventLDA achieved a precision of 0.418 and coverage of 0.340 with k=10 topics, outperforming wordLDA (k=5) in coverage.

Limitations and Future Work

  • Dependency Analyzer Dependence: The performance of eventLDA is dependent on the accuracy of the dependency structure analyzer used for event extraction. The paper uses a Japanese analyzer, implying potential language-specific limitations.
  • Topic Distribution Characteristics: The authors note that there is still space for examining the characteristics of event-based topic distributions as probabilistic distributions, especially given the unexpected performance of cosine similarity.
  • Future Work: The authors plan to conduct experiments with various data types and queries, and further investigate the characteristics of their proposed method. They also suggest exploring other features beyond words and events for latent topic estimation.

Summary updated

BibTeX
@inproceedings{kitajima2011latent,
  author = {Risa Kitajima and Ichiro Kobayashi},
  title = {{A Latent Topic Extracting Method based on Events in a Document and its Application}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2011},
}
Figure