Hierarchical MixUp Multi-label Classification with Imbalanced Interdisciplinary Research Proposals

Meng Xiao, Min Wu, Ziyue Qiao, Zhiyuan Ning, Yi Du, Yanjie Fu, Yuanchun ZhouarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces H-MixUp, a novel framework for classifying interdisciplinary research proposals. It tackles the challenge of imbalanced datasets and the hierarchical nature of research disciplines by combining transformer-based semantic extraction with a hierarchical MixUp strategy, significantly improving classification performance on interdisciplinary data.

What Problem Does This Paper Address?

Funding agencies rely on accurately matching research proposals to domain experts for review. However, as research becomes increasingly interdisciplinary, classifying these proposals poses a significant challenge. This is due to three key issues: 1) the hierarchical structure of discipline labels (e.g., from "Information Science" to "Artificial Intelligence" to "Fundamentals of AI"), 2) the heterogeneous nature of different proposal sections (title, abstract, etc.) which carry distinct semantic weight, and 3) a severe imbalance between non-interdisciplinary (NIR) and interdisciplinary (IR) research proposals, with IR proposals being a minority. Existing methods struggle to address all these complexities simultaneously, leading to models that are less effective at identifying and categorizing interdisciplinary research.

Methods

The proposed H-MixUp framework addresses these challenges through a multi-pronged approach:

  • Hierarchical Interdisciplinary Research Proposal Classification Network (HIRPCN): This forms the backbone of the system, processing proposals in a level-wise manner. It comprises three main components:
  • Semantic Information Extractor (SIE): Utilizes a hierarchical Transformer structure to extract semantic information from different parts of the proposal (title, abstract, etc.). This component incorporates four H-MixUp strategies:
  • Word-level MixUp: Mixes word embeddings before the Word-level Transformer.
  • Word-level CutMix: Similar to Word-level MixUp but uses the CutMix technique.
  • Manifold MixUp: Applies MixUp at a randomly selected layer within the SIE.
  • Document-level MixUp: Mixes document representations after the Word-level Transformer.
  • Interdisciplinary Knowledge Extractor (IKE): Employs Graph Convolutional Networks (GCNs) to embed previous prediction results and leverage a pre-built interdisciplinary graph, capturing relationships between disciplines.
  • Information Fusion (IF): Combines the mixed semantic information from SIE and the interdisciplinary knowledge from IKE using Multi-Head Self-Attention.
  • H-MixUp Strategy: The core innovation is the application of MixUp not just to input features but also to previous prediction results. This is done in a hierarchical, step-by-step manner, mixing representations from two different proposals at each level. This strategy is designed to mitigate the data imbalance issue by creating augmented, mixed samples that help the model learn more robust representations.

Datasets

The authors utilized datasets derived from the National Natural Science Foundation of China (NSFC) research funding application platform.

  • NSFC-All: This dataset contains all research proposals, exhibiting a significant imbalance between NIR and IR proposals (7% IR). It's used to train models and evaluate overall performance on imbalanced data.
  • NSFC-IR: This dataset is a subset of NSFC-All, containing only the IR proposals from the test set. It's used to specifically evaluate the model's performance on interdisciplinary data and highlight the impact of the imbalance.

Key Insights and Ideas

  • MixUp for Hierarchical Imbalance: The paper demonstrates that MixUp, traditionally used for image and text classification, can be effectively adapted to address hierarchical multi-label classification problems with imbalanced data.
  • Word-level MixUp is Key: Among the proposed H-MixUp strategies, Word-level MixUp consistently achieved the best performance, suggesting that mixing at the earlier stages of feature extraction is most beneficial for capturing interdisciplinary signals.
  • Data Imbalance Severely Degrades Performance: The experiments clearly show that the imbalance between IR and NIR data significantly degrades the performance of all models, especially in terms of Recall, leading to incomplete label path predictions.
  • H-MixUp Alleviates Imbalance: The H-MixUp strategy demonstrably improves model generalization and training stability on imbalanced datasets, leading to better performance compared to the original HIRPCN.
  • Attention Mechanism Benefits from MixUp: Visualizations of the attention mechanism show that H-MixUp, particularly Word-level MixUp, leads to more balanced attention on domain-related keywords, indicating a better understanding of the proposal's interdisciplinary nature.

Key Results

  • H-MixUp improved the SOTA performance by 14% on the imbalanced dataset when categorizing interdisciplinary data.
  • Compared to the original HIRPCN, H-MixUp strategies showed a lower performance deterioration (9.8% to 14.6%) when tested on the imbalanced NSFC-IR dataset compared to NSFC-All.
  • Word-level MixUp achieved the best F1 score on the NSFC-All dataset, outperforming other H-MixUp strategies and the original HIRPCN.
  • The study found that Word-level CutMix performed worse than other MixUp strategies, suggesting that concatenating text for mixing can lead to a significant loss of semantic information.

Limitations and Future Work

  • Word-level CutMix Performance: The paper notes that Word-level CutMix performed worse than other H-MixUp strategies, potentially due to the concatenation of text leading to a loss of semantic information.
  • Hyperparameter Sensitivity: The study conducted a hyperparameter study for the Beta distribution's alpha parameter, indicating that its selection can impact performance.
  • Future Work: While not explicitly detailed, the success of H-MixUp suggests future work could explore its application to other hierarchical multi-label classification tasks with imbalanced data. Further investigation into the optimal MixUp strategies for different types of textual data and hierarchical structures could also be beneficial.

Summary updated

Abstract

Funding agencies are largely relied on a topic matching between domain experts and research proposals to assign proposal reviewers. As proposals are increasingly interdisciplinary, it is challenging to profile the interdisciplinary nature of a proposal, and, thereafter, find expert reviewers with an appropriate set of expertise. An essential step in solving this challenge is to accurately model and classify the interdisciplinary labels of a proposal. Existing methodological and application-related literature, such as textual classification and proposal classification, are insufficient in jointly addressing the three key unique issues introduced by interdisciplinary proposal data: 1) the hierarchical structure of discipline labels of a proposal from coarse-grain to fine-grain, e.g., from information science to AI to fundamentals of AI. 2) the heterogeneous semantics of various main textual parts that play different roles in a proposal; 3) the number of proposals is imbalanced between non-interdisciplinary and interdisciplinary research. Can we simultaneously address the three issues in understanding the proposal's interdisciplinary nature? In response to this question, we propose a hierarchical mixup multiple-label classification framework, which we called H-MixUp. H-MixUp leverages a transformer-based semantic information extractor and a GCN-based interdisciplinary knowledge extractor for the first and second issues. H-MixUp develops a fused training method of Wold-level MixUp, Word-level CutMix, Manifold MixUp, and Document-level MixUp to address the third issue.
BibTeX
@misc{xiao2022hierarchical,
  author = {Meng Xiao and Min Wu and Ziyue Qiao and Zhiyuan Ning and Yi Du and Yanjie Fu and Yuanchun Zhou},
  title = {{Hierarchical MixUp Multi-label Classification with Imbalanced Interdisciplinary Research Proposals}},
  year = {2022},
  eprint = {2209.13912},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.13912},
}
Figure