Abstract
Rahul Jha, Amjad Abu-Jbara, Dragomir Radev. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
2013.
Introduction
As the volume of scientific publications explodes, researchers face a "deluge of data" that makes mastering new sub-fields increasingly difficult. While expert-written surveys are invaluable, they are rare and quickly become obsolete. This paper presents a pipeline that transforms a simple keyword query into a comprehensive scientific survey by combining a novel document retrieval method with graph-based summarization of citing sentences.
Methodology
The system addresses two primary challenges: identifying the "canon" of a field despite evolving terminology and extracting critical "factoids" to build an educational summary.
Candidate Document Selection: Restricted Expansion
Standard keyword searches often miss seminal early papers due to shifts in terminology (e.g., "Word Sense Disambiguation" vs. "Lexical Ambiguity Resolution"). To overcome this, the authors propose Restricted Expansion. The process begins with a high-precision "base set" of papers containing the exact query. This set is then expanded via the citation network—including papers that cite or are cited by the base set—and subsequently filtered to retain only the most central, well-cited documents.
Content Selection: Summarizing Citations
Rather than summarizing full texts, the system focuses on citing sentences. These snippets provide a distilled, third-party perspective on a paper's contributions. Three graph-based models were evaluated:
- Centroid: Selects sentences similar to the document cluster's average theme.
- Lexrank: Uses eigenvector centrality to find the most representative sentences.
- C-Lexrank: A variation of Lexrank designed to increase diversity through initial clustering.
Evaluation Framework
The authors utilized the ACL Anthology Network (AAN) as their primary corpus. To evaluate the system, they introduced a Factoid Inventory, manually extracting information units from 47 gold-standard surveys and tutorials across seven NLP topics. They annotated 2,625 sentences to determine which factoids they contained, enabling evaluation via the Pyramid Method, a robust metric for information content.
Results and Discussion
The experimental results validate the effectiveness of the proposed pipeline across both retrieval and summarization stages.
Retrieval Performance
The "Restricted Expansion" method significantly outperformed traditional retrieval techniques. As shown in Figure 2, it achieved a Cumulative Gain ($CG_{20}$) of 6.01, dwarfing the 0.56 score of standard TF*IDF ranking. This confirms that citation networks are a more reliable bridge across shifting terminologies than simple word-matching.
Summarization Quality
In the content selection phase, Lexrank emerged as the superior model. It achieved an average Pyramid score of 0.81, indicating a high degree of overlap with human-selected information units. Interestingly, even a random selection of citing sentences performed reasonably well (0.72), suggesting that the document selection step is so effective that most sentences from the retrieved "top papers" are informative.
Key Insights
- Citations as Summarizers: The scientific community effectively summarizes itself. Citing sentences act as pre-packaged "nuggets" of information, often superior to abstracts for building surveys.
- The Diversity Challenge: C-Lexrank underperformed compared to standard Lexrank. The authors hypothesize that the high lexical variation in scientific language makes it difficult for simple similarity metrics to identify redundant factoids.
Conclusion and Future Work
While the system successfully identifies high-quality information, several challenges remain. The current output can be redundant and lacks the narrative "connective tissue" of human-written surveys. Furthermore, citing sentences often lack basic definitions. Future work will explore integrating full-text analysis to extract introductory definitions and improving the overall coherence and flow of the generated summaries.
Summary updated