TL;DR
This paper explores the rich information contained within the "citing sentences" – the specific sentences in a research paper that reference another work. By analyzing these sentences within the ACL Anthology Network, the authors demonstrate their utility in understanding research trends, tracking the impact of scientific contributions over time, and even summarizing key discoveries.
What Problem Does This Paper Address?
Traditional bibliometric analysis often relies on simple citation counts to gauge the impact and importance of research. However, this approach overlooks the nuanced information embedded in the text surrounding citations. This paper addresses the gap by proposing and demonstrating how "citing sentences" can provide a deeper, more qualitative understanding of research dynamics, impact, and evolution within the field of Computational Linguistics (CL). The authors aim to show that these sentences offer more than just a link between papers; they are a valuable source of information for understanding the "why" and "how" of scientific progress.
Methods
The core of this paper's methodology revolves around the analysis of citing sentences extracted from the ACL Anthology Network (AAN). The AAN is a manually curated database of citations and collaborations within CL. The authors leverage this existing resource and propose several analytical approaches:
- Temporal Analysis of Citation Purpose and Polarity: They annotate citing sentences to determine the purpose of a citation (e.g., basis, comparison, weakness) and its polarity (positive, negative, neutral). By tracking these over time, they can analyze how a paper's reception evolves.
- Predicting Emergence of New Techniques: By observing shifts in citation purpose and polarity, they suggest it's possible to predict when new techniques might emerge or older ones decline in impact.
- Studying Research Dynamics: They propose using citing sentences, rather than just abstracts, to analyze research trends, contributions, and the interaction between research communities. This is based on the intuition that citing sentences offer a more focused and peer-validated perspective.
- Scientific Literature Summarization: The paper highlights how citing sentences can be used to create concise and informative summaries of research papers, often capturing contributions not present in the abstract.
- Controversy Identification: They suggest that citing sentences can reveal disputes or criticisms of previous work.
- Comparison of Techniques: Citing sentences often explicitly compare different methods, providing a valuable resource for researchers to quickly assess technique effectiveness.
- Ontology Creation: By extracting relationships between tasks, methods, and tools mentioned in citing sentences, they propose building a structured ontology.
- Paraphrase Extraction: The observation that different authors describe the same cited paper facts using varied wording motivates using citing sentences to create paraphrase datasets.
- Scientific Article Classification: They hypothesize that incorporating citing sentence text alongside citation structure and content can improve the accuracy of classifying scientific articles.
- Terminology Translation: By aligning citing sentences from papers in different languages that cite a common target paper, they propose a method for building parallel corpora of terms.
Datasets
- ACL Anthology Network (AAN): This is the primary dataset used throughout the paper. It's described as a comprehensive, manually curated networked database of citations and collaborations in the field of Computational Linguistics, containing citing sentences associated with citation links.
- Annotated Citing Sentences: For specific analyses, the authors performed manual annotation of citing sentences for:
- Citation Purpose: Based on Teufel et al.'s scheme (Table 2).
- Citation Polarity: For approximately 2000 citing sentences, and also referencing a study by Athar (2011) which annotated 8736 citations from AAN.
- Benchmark Datasets: The paper references existing work that used datasets for paraphrase extraction and summarization, often evaluated using AAN.
Key Insights and Ideas
- Citing Sentences as a Richer Source of Information: The central idea is that citing sentences are not just pointers but contain valuable, nuanced information about research impact, trends, and contributions that simple citation counts miss.
- Peer Validation in Citing Sentences: Contributions highlighted in citing sentences are "collectively deemed to be important by peer researchers," offering a more objective view than an author's self-description in an abstract.
- Predictive Power of Citation Dynamics: Changes in citation purpose and polarity over time can signal the emergence of new techniques or the decline of older ones, offering a predictive lens on research evolution.
- Summarization Beyond Abstracts: Citing sentences can provide more concise and informative summaries than abstracts, often including critical perspectives.
- Identifying Research Controversies: Citing sentences can explicitly highlight disputes and criticisms, offering a way to automatically identify areas of scientific disagreement.
Key Results
- Citation Polarity Distribution: The authors found that in their annotated sample, only 30% of citations were positive, 4.3% negative, and the rest neutral. A study by Athar (2011) on a larger AAN subset reported 10% positive, 3% negative, and the rest neutral. This highlights the importance of considering polarity beyond simple counts.
- Keyword Rank Shifts: Analysis of word frequencies in citing sentences across different decades (1980s, 1990s, 2000s) revealed shifts in research focus. For example, "grammar" was more prominent in the 1980s, and rule-based methods declined in popularity by the 2000s.
- Trigger Word Frequency: Trigger words indicative of paper characteristics appeared more frequently in citing sentences than in abstracts, suggesting that citing sentences contain more descriptive information.
- Paraphrase Extraction Data: An initial experiment on 25 papers from AAN yielded 33,683 sentence pairs, of which 8,704 were identified as paraphrases, demonstrating the potential for creating paraphrase datasets.
- ACL Summary: The paper presents a citation-based summary of important ACL contributions from 1979 onwards, created by selecting a representative citing sentence for the top-cited paper each year (Table 6).
Limitations and Future Work
- Annotation Effort: Manual annotation of citing sentences for purpose and polarity is labor-intensive. Future work could focus on developing more automated methods for this.
- Scope of Analysis: While the paper demonstrates various applications, further exploration and validation of these ideas across different research fields are suggested.
- Coherence and Readability: The paper acknowledges the importance of coherence and readability in summaries generated from citing sentences, with ongoing work in preprocessing and postprocessing steps.
- Automated Techniques: The authors propose several applications that would benefit from further development of automated techniques, such as relation extraction, named entity recognition, and improved indexing/ranking.
- Cross-Disciplinary Application: The paper focuses on CL, but the proposed methods for analyzing citing sentences could be applied to other scientific domains.
Summary updated