TL;DR
This paper presents Amrita_CEN_NLP's submission to the 3C Citation Context Classification shared task, aiming to understand the purpose and influence of citations in research papers. They tackle class imbalance using a cost-sensitive learning approach and achieve strong results with a Bi-LSTM model, outperforming other approaches on both subtasks.
What Problem Does This Paper Address?
In the vast landscape of scientific literature, understanding the impact and role of citations is crucial for assessing research quality. Traditional citation analysis often relies on simple citation counts, which can be misleading. This paper addresses the need for a more nuanced understanding by focusing on two key aspects: the purpose of a citation (how the cited work relates to the citing work) and its influence (whether the citation is incidental or genuinely impactful). The 3C shared task aims to automate this classification, and this paper details a team's approach to solving these classification problems. A significant challenge in this domain is the inherent class imbalance in citation datasets, where certain categories are far more prevalent than others.
Methods
The Amrita_CEN_NLP team employed a combination of machine learning and deep learning techniques. For feature extraction, they utilized TF-IDF with Singular Value Decomposition (SVD) for dimensionality reduction, along with fastText embeddings.
For classification, they experimented with two primary models:
- Random Forest: This traditional machine learning model was used in conjunction with the extracted features.
- Bi-LSTM (Bidirectional Long Short-Term Memory): This deep learning model was applied to the text data. The workflow involved tokenizing text, handling out-of-vocabulary words, converting words to integer indices, padding sequences, and using a one-hot encoded representation for labels. A softmax layer was used for the final classification.
A key methodological aspect was addressing the class imbalance problem. For the Random Forest model, they implemented a cost-sensitive learning approach. For the Bi-LSTM, they explored techniques like dropout and class weighting.
Datasets
The paper uses data from the 3C Citation Context Classification shared task at the Second Workshop on Scholarly Document Processing.
- Training Data: Provided via Kaggle, containing 3000 instances.
- Test Data: Contains 1000 datapoints.
The dataset includes fields such as citation context, cited paper title, citing paper title, and class labels. The authors highlight that the training data for Subtask-A (Purpose) is highly imbalanced, with the 'BACKGROUND' class comprising 54.93% of the data and 'FUTURE' only 2.07%. Subtask-B (Influence) data is described as balanced.
Key Insights and Ideas
- Beyond Citation Counts: The paper reinforces the idea that simply counting citations is insufficient for understanding scientific impact. The context and purpose of a citation are critical.
- Deep Learning for Context: Bi-LSTM models, with their ability to process sequential data and capture contextual information, show promise for nuanced citation analysis.
- Addressing Class Imbalance is Crucial: The authors explicitly tackled class imbalance, recognizing its significant impact on model performance, especially for the 'Purpose' subtask. Cost-sensitive learning and class weighting were key strategies.
- Hybrid Approaches Can Be Effective: While Bi-LSTM showed strong performance, the exploration of Random Forest with engineered features suggests that different approaches might excel under different conditions or for specific subtasks.
Key Results
The paper reports the performance of their submitted models for both subtasks.
- Subtask-A (Purpose): The Bi-LSTM + Dropout + Class Weight + 128 hidden neurons model achieved the best performance on both private and public scores.
- Subtask-B (Influence): While the Random Forest with fastText embedding achieved the highest private score, the Bi-LSTM + Dropout + Class Weight + 64 hidden neurons model scored the maximum in the public score.
The paper provides tables (Table 4 and Table 5) detailing these private and public scores for various model configurations.
Limitations and Future Work
- Limitations: The paper does not explicitly detail limitations beyond the inherent class imbalance of the dataset. The performance differences between private and public scores suggest potential overfitting or variations in the test set distribution.
- Future Work: The authors do not explicitly outline future work directions in the provided text. However, the success of the Bi-LSTM model suggests further exploration of deep learning architectures and attention mechanisms for citation context classification would be a natural next step. Investigating more sophisticated methods for handling class imbalance or exploring transfer learning from larger text corpora could also be promising avenues.
Summary updated