Attribute Extraction from Conjectural Queries

Marius PaşcaCOLING 2012

TL;DR

This paper introduces a novel method for extracting "conjectural attributes" – properties users inquire about in search queries (e.g., "is millennium stadium heated?") – that are often missing from traditional knowledge bases. By analyzing these user-generated queries, the authors demonstrate a way to enrich structured knowledge with user-centric information, achieving high accuracy in attribute extraction.

What Problem Does This Paper Address?

Building comprehensive knowledge bases (like Wikipedia or Freebase) is a monumental task. While these resources contain a wealth of information about entities and their properties, they often lack attributes that are of genuine interest to users. These missing attributes might be subjective (e.g., "is this movie funny?") or simply not yet documented. Traditional information extraction methods often struggle to capture these "conjectural" attributes because they aren't always expressed in a standard, nominal form. The authors aim to bridge this gap by leveraging the vast, often overlooked, data source of web search queries to discover and extract these valuable, user-driven attributes.

Methods

The core of the paper's contribution lies in its method for identifying and extracting "conjectural attributes" from web search queries. The approach focuses on two main types of queries:

  • Conjectural Queries: These are truth-verification queries, typically in the form of "is X Y?" or "does X have Y?". For example, "is millennium stadium heated?". The presence of such queries suggests that "heated" is a potential attribute for "millennium stadium" and, by extension, for the class "Stadiums."
  • Explanation-Seeking Queries: These queries, like "why is X Y?", are considered more reliable indicators of an attribute's relevance. They imply that the attribute is already known or assumed to be true, and the user is seeking an explanation.

The method involves:

  1. Identifying Target Classes and Instances: Using Wikipedia categories as target classes and extracting instances from them.
  2. Query Pattern Matching: Identifying queries that fit the "is X Y?" or "why is X Y?" patterns, where X is an instance and Y is a potential attribute.
  3. Instance Disambiguation: Matching query instances to their corresponding Wikipedia entries.
  4. Attribute Aggregation: Collecting potential attributes for instances and then aggregating them to the class level. Attributes that appear for fewer than five distinct instances of a class are filtered out as unreliable.
  5. Ranking: Attributes are ranked based on how many instances of a class they apply to and their frequency in query logs.

Datasets

  • Web Search Queries: A random sample of approximately 500 million fully anonymized English web search queries were used. Each query was accompanied by its frequency of occurrence.
  • Wikipedia Categories: 40 Wikipedia categories were used as target classes. These were mapped from existing experimental setups and represent diverse domains.
  • Wikipedia Articles: Articles within these categories (and their sub-categories) were used to define the instances for each target class.

Key Insights and Ideas

  • User Queries as a Rich Source of Missing Knowledge: The paper powerfully demonstrates that the collective "wisdom of the crowds" expressed through search queries can reveal attributes that are not readily available in curated knowledge bases.
  • Conjectural Queries are Valuable, Even if "Weak": While "is X Y?" queries provide weaker evidence than "why X Y?" queries, their sheer volume makes them a significant data source for discovering attributes.
  • Subjective Attributes are Discoverable: The method can extract not only objective attributes (e.g., "case sensitive" for programming languages) but also subjective ones (e.g., "funny" for films), reflecting a broader spectrum of user interest.
  • Complementarity to Existing Knowledge: Conjectural attributes are shown to be distinct from and complementary to attributes found in Wikipedia, Freebase, or extracted by previous methods, often capturing properties that are more user-centric or less formally defined.

Key Results

  • High Accuracy: The proposed method achieved an average precision score of 0.84 across 40 target classes.
  • Class-Specific Performance: Accuracy varied by class, with Programming Languages achieving a high score of 0.98, while Wine scored 0.46.
  • Attribute Diversity: The extracted attributes cover a wide range of linguistic forms, from single adjectives (e.g., "heated?") to multi-word phrases (e.g., "a retractable roof?").
  • Mapping to Nominal Attributes: A significant portion (72%) of conjectural attributes were found to have potential mappings to existing nominal attributes, with 28% preserving meaning fully and 44% approximating it. However, a substantial portion (31% + 28%) did not have readily available nominal equivalents, highlighting their novelty.

Limitations and Future Work

  • Noisy Instances: The presence of noisy or incorrectly disambiguated instances within Wikipedia categories can lead to the extraction of attributes that are relevant to individual instances but not to the class as a whole.
  • Ambiguity in Query Interpretation: While the method uses pattern matching, the inherent ambiguity in natural language queries can still pose challenges.
  • Future Work: The authors suggest investigating the role of repositories of class labels extracted from text and exploring the acquisition of (mostly binary) values for these conjectural attributes for individual instances. They also note the potential for filtering noisy instances to improve accuracy.

Summary updated

BibTeX
@inproceedings{pasca2012attribute,
  author = {Marius Paşca},
  title = {{Attribute Extraction from Conjectural Queries}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure