A Method for Automatically Generating a Mediatory Summary to Verify Credibility of Information on the Web

Hideyuki Shibuki, Takahiro Nagai, Masahiro Nakano, Rintaro Miyazaki, Madoka Ishioroshi, Tatsunori MoriCOLING 2010

TL;DR

This paper introduces a novel method for automatically generating "mediatory summaries" from web documents. These summaries aim to help users assess the credibility of online information by presenting balanced perspectives and highlighting potential conflicts, going beyond what traditional search engines offer.

What Problem Does This Paper Address?

The internet is awash with information, but not all of it is reliable. Distinguishing credible information from misinformation is a significant challenge for users. While search engines help retrieve documents, they don't inherently assist in evaluating the credibility of that information, especially when conflicting viewpoints exist across multiple documents. Existing methods often focus on individual document credibility or semantic relationships, but there's a need for a tool that can synthesize conflicting information from various sources into a digestible format to aid user judgment.

Methods

The core of this paper is the development of an algorithm to automatically generate "mediatory summaries." This process involves several key steps:

  • Inverse Query Generation: To find potentially conflicting information, the system generates "inverse queries" by replacing words in the original query with their antonyms. For example, if the query is "Is safety of LASIK operation high?", inverse queries like "Is risk of LASIK operation high?" or "Is safety of LASIK operation low?" are created.
  • Web Document Retrieval: The original query and its inverse queries are used to retrieve distinct sets of web documents. Documents retrieved by the original query are likely to contain positive viewpoints, while those from inverse queries are expected to contain negative viewpoints.
  • Keyword Extraction: The system identifies "topic keywords" from the original query, and then extracts "positive keywords" and "negative keywords" from the retrieved documents. These keywords are crucial for identifying balanced and potentially conflicting content. The extraction process uses scores (SCPOS and SCNEG) that consider word frequency in both query-specific and inverse-query-specific document sets.
  • Passage Extraction: This is the most complex stage. Sentences are first filtered for usability. Then, a "basic score" (SCBAS) is calculated for each sentence based on the presence of topic, positive, and negative keywords. This score is penalized for "useless" sentences. A "smoothed score" (SCSMO) is then computed using a Hann window to consider sentence context within a passage. Finally, passages are extracted and ranked based on their highest sentence score (SCPAS) and their length relative to an ideal length (CLength), resulting in a final score (SCFIN).

The mediatory summary itself is a passage extracted from web documents, augmented with italicized text generated through templates to guide the user.

Datasets

The paper mentions the use of a "text summarization corpus" constructed by Nakano et al. (2010) for determining web information credibility. This corpus contains query statements, retrieved web documents, and human-generated summaries. For their experiments, they used six specific queries: "Are diesel engines harmful to the environment?", "Is safety of LASIK operation high?", "Are whales endangered species?", "Does asbestos have toxics?", "Is catch and release a better way of fishing?", and "Does carbon dioxide cause global warming?". They also utilized the Japanese morphological analyzer, MeCab.

Key Insights and Ideas

  • The Importance of "Fairness": The paper strongly emphasizes "fairness" as a key feature for mediatory summaries. This means presenting both positive and negative viewpoints, not just summarizing one side. The proposed method's success is partly attributed to its ability to identify and balance these opposing perspectives.
  • Beyond Simple Keyword Matching: The method goes beyond just finding frequent words. By categorizing keywords into "topic," "positive," and "negative," and by using inverse queries, the system can more effectively identify content that directly addresses conflicting aspects of a query.
  • Context Matters for Summaries: The passage extraction method, particularly the smoothing stage, highlights that the context of a sentence within a passage is crucial for its contribution to a mediatory summary. A single sentence, even if informative, might be less useful if it's isolated.
  • Mediatory Summaries are a Distinct Task: The comparison with general-purpose summarization tools (LinTsu) suggests that simply extracting characteristic sentences isn't enough. Mediatory summaries require a specific approach to handle conflicting information and present it in a way that facilitates user judgment.

Key Results

The proposed method demonstrated superior performance compared to baseline methods (KWtf, LinTsu, LinscFIN) in generating appropriate mediatory summaries.

  • High Precision: Across multiple queries and for different numbers of top summaries (Top 3, Top 5, Top 10), the proposed method consistently achieved the highest average precision scores. For instance, for the query "Are diesel engines harmful to the environment?", the proposed method achieved 100.0% precision for the top 3 summaries, significantly outperforming other methods.
  • Fairness Contributes: The difference in results between the proposed method and KWtf (which doesn't consider keyword polarity) indicates that classifying keywords into positive and negative is crucial for generating fair and appropriate mediatory summaries.
  • General Summarization is Insufficient: The comparison with LinTsu showed that general-purpose summarization tools, while good at extracting characteristic sentences, are not as effective for the specific task of creating mediatory summaries that address conflicting viewpoints.

Limitations and Future Work

  • Parameter Tuning: The paper acknowledges that the parameters used in the method (e.g., Crank, Cdif, Cuseless, Cbasic, Csmooth, Cpassage, Clength, a) were set tentatively based on preliminary experiments. Determining optimal parameters is a topic for future research.
  • Limited Coverage on Some Queries: For the queries "Is catch and release a better way of fishing?" and "Does carbon dioxide cause global warming?", the proposed method did not extract all appropriate passages that were identified by the LinTsu method. The authors aim to improve the method to address this.
  • Topic Keyword Definition: Currently, topic keywords are restricted to words within the given query. The paper suggests that including words other than those in the query could be a direction for future research.
  • Template Generation: The paper notes that the discussion on generation through templates was omitted due to space limitations, implying this is an area that could be further explored.

Summary updated

BibTeX
@inproceedings{shibuki2010method,
  author = {Hideyuki Shibuki and Takahiro Nagai and Masahiro Nakano and Rintaro Miyazaki and Madoka Ishioroshi and Tatsunori Mori},
  title = {{A Method for Automatically Generating a Mediatory Summary to Verify Credibility of Information on the Web}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure