Measuring Term Informativeness in Context

Zhaohui Wu, C. Lee GilesNAACL 2013

TL;DR

This paper introduces a novel, feature-free method called Context-aware Term Informativeness (CTI) to measure how informative a word is within its specific context. By leveraging web knowledge, particularly Wikipedia, CTI models the semantic relatedness between a term's context and its most featured contexts, offering a more nuanced understanding than traditional statistical methods. This approach demonstrates effectiveness across diverse NLP tasks like core term extraction, keyword extraction, and index generation.

What Problem Does This Paper Address?

Measuring the "informativeness" of a word is crucial for many Natural Language Processing (NLP) tasks, such as keyword extraction, text summarization, and document clustering. Traditional methods often rely on statistical measures like Term Frequency-Inverse Document Frequency (TF-IDF) or document frequency. While effective for large corpora, these methods can struggle with smaller texts and, more importantly, fail to capture the semantic meaning of a word within its specific context. For instance, statistical methods might not distinguish the importance of "graph" in a sentence about graph theory versus its importance in a sentence about social networks. The authors argue that existing methods are often ad hoc, require manual feature engineering, and don't truly measure a term's informativeness relative to its discourse. They aim to bridge this gap by proposing a context-aware, semantic-based approach that leverages external knowledge.

Methods

The core of the paper is the Context-aware Term Informativeness (CTI) metric. The approach involves several key steps:

  1. Context Definition: A "context" is defined as a syntactic unit of discourse, such as a sentence or paragraph.
  2. Context Authority: The "authority" of a context is estimated by its popularity on the web (e.g., number of search engine results). This is a proxy for how important or representative that context is for a given term.
  3. Featured Context Set (Uf): For a given term, the top k contexts with the highest authority scores are selected. This set is assumed to capture the most important semantic aspects of the term.
  4. Semantic Relatedness: A semantic relatedness metric (e.g., Wikipedia-based similarity) is used to measure how closely related two contexts are.
  5. CTI Calculation: The informativeness of a term t in a context ci is calculated as a weighted sum of the semantic relatedness between ci and the contexts in its featured context set (Uf). The formula is: $I(t, C_i) = \sum_{c_j \in U_f(t)} \kappa(C_i, C_j) \cdot CA(C_j)$, where $\kappa$ is the semantic relatedness and $CA$ is the context authority.

The implementation leverages Wikipedia as the knowledge base, using its page titles and content to derive featured contexts and semantic relatedness. The context authority is estimated by the rank of the Wikipedia document.

Datasets

The paper evaluates CTI on several existing datasets and implicitly uses web search results as a source of authority:

  • Descriptive Snippets: A collection of 1255 computer science terms with definition snippets, manually created by web users. This dataset is used for core term extraction.
  • Wiki20: A dataset of 20 computer science articles with approximately 5 terms each, labeled by 15 different teams. Used for keyword extraction.
  • citeulike180: A dataset of 180 papers tagged with around three tags by 332 users. Used for keyword extraction.
  • SemEval2010 Task 5: A dataset of 284 scientific papers with 15 keyphrases assigned by readers and authors, split into training and testing sets. Used for keyword extraction.
  • Gutenberg Dataset: 55 free books collected from Project Gutenberg, used for back-of-the-book index generation (evaluated as keyword extraction due to lack of locators).
  • Open Book Dataset: A collection of computer science and engineering books from CiteSeer, used for back-of-the-book index generation, which includes locators.

Key Insights and Ideas

  • Context is King (Semantically): The paper strongly advocates that a term's informativeness is highly dependent on its surrounding text and that semantic understanding is key to capturing this.
  • Web Knowledge as a Proxy for Semantics: Leveraging large, publicly available knowledge bases like Wikipedia provides a scalable way to access semantic information and context authority without needing domain-specific corpora or manual feature engineering.
  • Featured Contexts Capture Essence: The idea that a small set of top-ranked contexts (the "featured context set") can sufficiently represent a term's semantic meaning is a powerful simplification.
  • Generality Across Tasks: The CTI metric proves to be adaptable and effective across diverse NLP applications, suggesting its fundamental nature.
  • Feature-Free Advantage: The CTI approach is "feature-free," meaning it doesn't require manual selection or engineering of linguistic or statistical features, making it more robust and easier to implement.

Key Results

The paper reports state-of-the-art or near state-of-the-art performance across its evaluated applications:

  • Core Term Extraction from Snippets: CTI achieved 37.5% precision for identifying the single core term, and recall increased to 76.3% when considering the top 5 results.
  • Keyword Extraction: CTI performed competitively on Wiki20 and citeulike180 datasets, sometimes outperforming or being comparable to established methods like KEA and TFIDF, especially when context is considered. On SemEval2010, CTI achieved 19.8% F1 score, improving to 25.7% when incorporating structural features.
  • Back-of-the-Book Index Generation: CTI outperformed other context-oblivious metrics on both Gutenberg and Open Book datasets for keyword extraction. When adapted for index generation (selecting term-context pairs), CTI showed promising results, with "CI-Indexer" (CTI with context weights) performing comparably to or better than baselines like TFIDF and KEA, especially when context was incorporated.

Limitations and Future Work

  • Context Definition: The paper treats paragraphs as individual contexts, acknowledging that this might not always be a complete discourse unit. Future work could explore more flexible context segmentation.
  • Knowledge Base Dependence: The effectiveness of CTI is tied to the quality and coverage of the knowledge base used (e.g., Wikipedia). Its performance on highly specialized or informal domains (like tweets or forum posts) might differ.
  • Semantic Relatedness Metric: The choice of semantic relatedness metric can influence results. Further investigation into optimal metrics is suggested.
  • Scalability of Web Search: While the method is inherently parallelizable, reliance on web search engines for context authority can be subject to network conditions and search engine result variability.
  • Future Work: The authors suggest exploring different context segmentation strategies, investigating other knowledge bases, and building practical tools like an online indexing tool that leverages CTI. They also highlight the need for further research into how different semantic relatedness metrics and knowledge bases affect performance.

Summary updated

BibTeX
@inproceedings{wu2013measuring,
  author = {Zhaohui Wu and C. Lee Giles},
  title = {{Measuring Term Informativeness in Context}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure