Reference Scope Identification for Citances Using Convolutional Neural Networks

Saurav Jha, Aanchal Chaurasia, Akhilesh Sudhakar, Anil Kumar SinghICON 2017 · OpenReview

Abstract

Reference Scope Identification is the task of identifying specific sentences in a reference paper that are cited by a given "citance" in a citing paper. This task is critical for automated scientific document summarization but is hindered by extreme class imbalance and small annotated datasets. This paper proposes a binary classification approach using a 1D Convolutional Neural Network (CNN) trained on a diverse set of lexical, semantic, and structural features. By incorporating techniques like SMOTE for oversampling and PCA for dimensionality reduction, the proposed model significantly outperforms previous state-of-the-art methods on the CL-SciSumm shared task datasets.

Introduction

In scientific writing, a "citance" refers to a sentence in a citing paper (CP) that references a specific reference paper (RP). However, citations rarely target an entire document; they usually point to specific ideas or methods contained within a few sentences, known as the "reference scope." Identifying this scope automatically is the primary objective of Task 1A in the Computational Linguistics Scientific Document Summarization (CL-SciSumm) shared task.

The challenge is primarily one of scale and imbalance: a single citance might refer to only one or two sentences out of hundreds in a reference paper, creating a "needle in a haystack" scenario. Accurate identification of these sentences is a prerequisite for high-quality automated summarization of scientific literature.

Methodology

The authors frame the problem as a binary classification task, pairing a citance with every sentence in the reference paper to determine if a citation relationship exists.

Feature Engineering

Given the limited size of annotated datasets (e.g., 495 positive instances), the authors found that training on extracted features was more effective than using raw text. They extracted four categories of features:

  • Lexical: Jaccard similarity, Cosine similarity, TF-IDF, and ROUGE metrics.
  • Knowledge-based: Semantic similarity via WordNet synonyms.
  • Corpus-based: Word2Vec embeddings for contextual meaning.
  • Surface: Structural details like word counts and special characters.

Model Architecture

The primary model is a 1D Convolutional Neural Network (CNN), which is adept at identifying local patterns within feature sequences. The architecture processes the feature vectors through convolutional and pooling layers before reaching a fully connected output layer.

Architecture of the 1D CNN model used for classification.

To address the 350:1 class imbalance, the authors employed the Synthetic Minority Over-sampling Technique (SMOTE). Additionally, Principal Component Analysis (PCA) was used to reduce noise and redundancy in the high-dimensional feature set.

Results and Discussion

The CNN-based approach established a new benchmark for the CL-SciSumm task. On the 2016 test set, the model achieved an F1-score of 0.2462, nearly doubling the performance of the previous state-of-the-art (0.1443). On a combined 2016/2017 dataset, the model reached an F1-score of 0.5558.

The Impact of Dimensionality Reduction and Embeddings

Ablation studies highlighted the critical role of PCA and semantic features. As shown in the figure below, removing PCA caused the F1-score to plummet from 0.5558 to 0.2838, indicating that the model's ability to generalize depends heavily on distilling the most predictive signals from the feature set.

Ablation study showing the impact of various techniques on model performance.

While lexical features (word overlap) remained the most influential individual category, the inclusion of Word2Vec embeddings provided a necessary boost by allowing the model to recognize synonyms and paraphrased concepts.

Conclusion

This research demonstrates that feature-based deep learning is a robust strategy for NLP tasks in niche domains with limited data. The 1D CNN, combined with strategic oversampling and dimensionality reduction, effectively navigates the extreme class imbalance inherent in reference scope identification. Future work will explore "Learning to Rank" frameworks and more advanced context representations like Doc2Vec to further refine the identification of cited content.

Summary updated

BibTeX
@inproceedings{jha2017reference,
  author = {Saurav Jha and Aanchal Chaurasia and Akhilesh Sudhakar and Anil Kumar Singh},
  title = {{Reference Scope Identification for Citances Using Convolutional Neural Networks}},
  booktitle = {ICON},
  year = {2017},
  url = {https://openreview.net/forum?id=B9jRUxnZ4JH},
}
Figure