Using the Annotated Bibliography as a Resource for Indicative Summarization

Min-Yen Kan, Judith L. Klavans, Kathleen R. McKeown

Abstract

This paper introduces the Annotated Bibliography Corpus, a novel dataset of 2,000 human-written entries designed to advance the field of indicative, abstractive summarization. Moving beyond traditional sentence extraction, the researchers analyze the structural "grammar" and metadata-rich features of annotated bibliographies. Their findings reveal a predictable strategic order in human summaries and provide a robust framework for training natural language generation (NLG) systems to produce summaries that help users evaluate document relevance.

Introduction

In the early 2000s, automatic text summarization primarily relied on extractive methods—algorithms that identified and concatenated key sentences from source texts. While effective for information retrieval, these extracts often lacked the coherence and nuance of human-authored summaries. Researchers at Columbia University identified a critical need for "indicative" summaries: concise, abstractive teasers that allow readers to quickly determine a document's utility. To address the lack of suitable training data for this task, the authors turned to the annotated bibliography, a resource rich in human-written evaluative and descriptive content.

The Annotated Bibliography Corpus

The researchers constructed a corpus of 2,000 entries by spidering the web for the phrase "annotated bibliography" using search engines like AltaVista and Google. These entries were sourced from 64 distinct web pages to ensure diversity. Each entry was encoded in XML to preserve its original context, including the "Before Context" (e.g., bibliographic metadata like title and author) and "Micro/Macro Collection" data, which situates the entry within a broader thematic framework.

Feature Taxonomy and Manual Annotation

To understand the underlying structure of these summaries, the team performed deep manual annotation on a subset of 100 entries. They identified 24 distinct document features—types of information provided to the reader—including:

  • Audience: Who the text is intended for.
  • Difficulty: The level of expertise required.
  • Subjective Assessment: The bibliographer's evaluation of the work.
  • Media Type: The format of the source material.

Additionally, the entries underwent linguistic processing via a probabilistic dependency parser to map the grammatical structures characteristic of human-written summaries.

Comparative Analysis

The Annotated Bibliography Corpus fills a gap left by existing benchmarks. While the Document Understanding Conference (DUC) datasets focus on informative, generic summaries and the Ziff-Davis corpus is largely extractive, annotated bibliographies provide high linguistic variation. Unlike scientific abstracts, which are often purely indicative, bibliographies include metadata like "purpose" and "audience," which are essential for search engine users. Because these summaries are not tied to the source text's exact wording, they exhibit higher variation in sentence structure and lexical choice, making them a superior "gold standard" for training natural language generation (NLG) systems.

Key Findings: The "Grammar" of Summaries

A significant insight from the study is that human-written indicative summaries follow a predictable "strategic order." For instance, an Overview sentence typically appears at the beginning of an entry (average position 0.10), whereas Subjective Assessments or Contributor details are usually relegated to the end.

Results and Feature Distribution

The data showed that most summaries are concise, typically containing between 2 and 6 distinct features. In the annotated subset:

  • Overview sentences were present in 64% of entries.
  • Detail (specific facts or conclusions) appeared in 47%.
  • Entries with more than 13 features were rare, accounting for only 6% of the corpus.

This consistency validates existing library science writing guides, suggesting they can serve as a formal framework for AI-driven summarization.

Conclusion and Future Directions

The Annotated Bibliography Corpus represents a shift toward more sophisticated, human-like summarization. While the manual annotation was limited by the labor-intensive nature of the task, the corpus provides a robust resource for training systems that go beyond simple extraction. Future work aims to utilize this data to develop "strategic components" for NLG, teaching models not only what information to include but also the optimal sequence for presenting it to a human reader.

Summary updated

Abstract

We report on a language resource consisting of 2000 annotated bibliography entries, which is being analyzed as part of our research on indicative document summarization. We show how annotated bibliographies cover certain aspects of summarization that have not been covered by other summary corpora, and motivate why it is an important form to study for information retrieval. We detail our methodology for collecting the corpus, and overview our document feature markup that we introduced to facilitate summary analysis. We present the characteristics of the corpus, methods of collection, and show its use in finding the distribution of types of information included in indicative summaries and their relative ordering within the summaries.
BibTeX
@inproceedings{kan2002using,
  author = {Min-Yen Kan and Judith L. Klavans and Kathleen R. McKeown},
  title = {{Using the Annotated Bibliography as a Resource for Indicative Summarization}},
  booktitle = {LREC},
  year = {2002},
  url = {https://openreview.net/forum?id=eCk9f7NzbE6},
}
Figure