TL;DR
This paper introduces H-MixUp, a novel framework for classifying interdisciplinary research proposals. It tackles the challenge of imbalanced datasets and the hierarchical nature of research disciplines by combining transformer-based semantic extraction with a hierarchical MixUp strategy, significantly improving classification performance on interdisciplinary data.
What Problem Does This Paper Address?
Funding agencies rely on accurately matching research proposals to domain experts for review. However, as research becomes increasingly interdisciplinary, classifying these proposals poses a significant challenge. This is due to three key issues: 1) the hierarchical structure of discipline labels (e.g., from "Information Science" to "Artificial Intelligence" to "Fundamentals of AI"), 2) the heterogeneous nature of different proposal sections (title, abstract, etc.) which carry distinct semantic weight, and 3) a severe imbalance between non-interdisciplinary (NIR) and interdisciplinary (IR) research proposals, with IR proposals being a minority. Existing methods struggle to address all these complexities simultaneously, leading to models that are less effective at identifying and categorizing interdisciplinary research.
Methods
The proposed H-MixUp framework addresses these challenges through a multi-pronged approach:
- Hierarchical Interdisciplinary Research Proposal Classification Network (HIRPCN): This forms the backbone of the system, processing proposals in a level-wise manner. It comprises three main components:
- Semantic Information Extractor (SIE): Utilizes a hierarchical Transformer structure to extract semantic information from different parts of the proposal (title, abstract, etc.). This component incorporates four H-MixUp strategies:
- Word-level MixUp: Mixes word embeddings before the Word-level Transformer.
- Word-level CutMix: Similar to Word-level MixUp but uses the CutMix technique.
- Manifold MixUp: Applies MixUp at a randomly selected layer within the SIE.
- Document-level MixUp: Mixes document representations after the Word-level Transformer.
- Interdisciplinary Knowledge Extractor (IKE): Employs Graph Convolutional Networks (GCNs) to embed previous prediction results and leverage a pre-built interdisciplinary graph, capturing relationships between disciplines.
- Information Fusion (IF): Combines the mixed semantic information from SIE and the interdisciplinary knowledge from IKE using Multi-Head Self-Attention.
- H-MixUp Strategy: The core innovation is the application of MixUp not just to input features but also to previous prediction results. This is done in a hierarchical, step-by-step manner, mixing representations from two different proposals at each level. This strategy is designed to mitigate the data imbalance issue by creating augmented, mixed samples that help the model learn more robust representations.
Datasets
The authors utilized datasets derived from the National Natural Science Foundation of China (NSFC) research funding application platform.
- NSFC-All: This dataset contains all research proposals, exhibiting a significant imbalance between NIR and IR proposals (7% IR). It's used to train models and evaluate overall performance on imbalanced data.
- NSFC-IR: This dataset is a subset of NSFC-All, containing only the IR proposals from the test set. It's used to specifically evaluate the model's performance on interdisciplinary data and highlight the impact of the imbalance.
Key Insights and Ideas
- MixUp for Hierarchical Imbalance: The paper demonstrates that MixUp, traditionally used for image and text classification, can be effectively adapted to address hierarchical multi-label classification problems with imbalanced data.
- Word-level MixUp is Key: Among the proposed H-MixUp strategies, Word-level MixUp consistently achieved the best performance, suggesting that mixing at the earlier stages of feature extraction is most beneficial for capturing interdisciplinary signals.
- Data Imbalance Severely Degrades Performance: The experiments clearly show that the imbalance between IR and NIR data significantly degrades the performance of all models, especially in terms of Recall, leading to incomplete label path predictions.
- H-MixUp Alleviates Imbalance: The H-MixUp strategy demonstrably improves model generalization and training stability on imbalanced datasets, leading to better performance compared to the original HIRPCN.
- Attention Mechanism Benefits from MixUp: Visualizations of the attention mechanism show that H-MixUp, particularly Word-level MixUp, leads to more balanced attention on domain-related keywords, indicating a better understanding of the proposal's interdisciplinary nature.
Key Results
- H-MixUp improved the SOTA performance by 14% on the imbalanced dataset when categorizing interdisciplinary data.
- Compared to the original HIRPCN, H-MixUp strategies showed a lower performance deterioration (9.8% to 14.6%) when tested on the imbalanced NSFC-IR dataset compared to NSFC-All.
- Word-level MixUp achieved the best F1 score on the NSFC-All dataset, outperforming other H-MixUp strategies and the original HIRPCN.
- The study found that Word-level CutMix performed worse than other MixUp strategies, suggesting that concatenating text for mixing can lead to a significant loss of semantic information.
Limitations and Future Work
- Word-level CutMix Performance: The paper notes that Word-level CutMix performed worse than other H-MixUp strategies, potentially due to the concatenation of text leading to a loss of semantic information.
- Hyperparameter Sensitivity: The study conducted a hyperparameter study for the Beta distribution's alpha parameter, indicating that its selection can impact performance.
- Future Work: While not explicitly detailed, the success of H-MixUp suggests future work could explore its application to other hierarchical multi-label classification tasks with imbalanced data. Further investigation into the optimal MixUp strategies for different types of textual data and hierarchical structures could also be beneficial.
Summary updated