Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training

Ran Xu, Yue Yu, Joyce C. Ho, Carl YangSIGIR 2023 · DOI · OpenReview

TL;DR

This paper introduces WANDER, a novel weakly-supervised method for classifying scientific documents using only label names. WANDER overcomes the challenge of domain-specific label names not appearing in documents by employing dense retrieval to capture semantic similarity, expanding label names with relevant keywords, and refining predictions through self-training.

What Problem Does This Paper Address?

Classifying scientific documents into predefined categories is crucial for many applications, such as literature search and knowledge discovery. However, obtaining large amounts of human-labeled data for training supervised models is expensive and time-consuming. While weakly-supervised methods that use only label names as supervision offer a promising alternative, they struggle with scientific domains. This is because scientific label names are often highly specialized and may not directly appear in the documents, leading to a "coverage gap." Existing methods that rely on keyword matching or simple embedding similarity fail to bridge this gap effectively, resulting in significantly lower performance compared to fully supervised approaches on scientific datasets.

Methods

WANDER tackles this problem through a three-stage training process:

  • Stage-I: Dense Retrieval with Label Names: Instead of relying on exact keyword matches, WANDER uses dense retrieval (DR) to find documents semantically similar to label names. This is achieved by encoding both label names (as queries) and documents into a shared embedding space. The DR model is first pre-trained using contrastive learning on the unlabeled corpus to improve its ability to align text representations. Then, label names are used as queries to retrieve a set of relevant documents from the unlabeled corpus. These retrieved documents, labeled according to their corresponding query label name, form an initial pseudo-labeled dataset.
  • Stage-II: Expand Label Names with Local and Global Information: To address the limited semantic coverage of short label names, WANDER expands them. It extracts candidate keywords from the retrieved documents using a TF-IDF-like approach (local information) and then scores these keywords based on their semantic similarity to the label name using a pre-trained language model (global information). These local and global scores are combined using Reciprocal Rank Fusion (RRF) to rerank the candidate keywords. The top-ranked keyword is then appended to the label name. This expansion process is iterated multiple times, with the updated label names used to retrieve a new set of documents, progressively enriching the label representations and improving retrieval quality.
  • Stage-III: Refine Classifier with Self-training: Finally, to leverage the entire unlabeled corpus and improve generalization, WANDER employs self-training. The classifier, fine-tuned on the pseudo-labeled data from Stage-II, is further trained on the unlabeled documents using its own high-confidence predictions as pseudo-labels. This iterative refinement process helps the model learn from a broader range of data.

Datasets

The paper evaluates WANDER on three scientific document classification datasets:

  • MeSH: Biomedical domain.
  • arXiv-Math: Mathematics domain.
  • arXiv-CS: Computer Science domain.

These datasets are characterized by their domain-specific terminology and imbalanced label distributions, making them challenging for existing weakly-supervised methods.

Key Insights and Ideas

  • Dense Retrieval is Key for Scientific Domains: The core insight is that dense retrieval, by capturing semantic similarity in an embedding space, is far more effective than traditional keyword matching for scientific documents where label names might not explicitly appear.
  • Label Name Expansion is Crucial: Simply using raw label names is insufficient. Expanding them with relevant keywords, derived from both local document context and global semantic understanding, significantly improves the model's ability to capture class-specific information.
  • Multi-stage Training Synergizes: Each stage of WANDER builds upon the previous one, creating a synergistic effect. Dense retrieval provides initial relevant data, label expansion refines the understanding of what's relevant, and self-training broadens the learning scope.
  • Scientific Domains Pose Unique Challenges: The paper highlights that methods successful on general domains (like news or reviews) often fail on scientific data due to the specialized nature of terminology and the resulting coverage gap.

Key Results

WANDER consistently outperforms existing weakly-supervised baselines across all three datasets, achieving an average gain of 11.9% in Macro-F1 score. For instance, on the MeSH dataset, WANDER achieved a Macro-F1 of 81.9%, significantly outperforming the best baseline (FastClass) at 78.1%. The paper also shows that each stage of WANDER contributes to the final performance, with Stage-I and Stage-II alone already yielding substantial improvements. The ablation studies confirm the effectiveness of dense retrieval pre-training and the complementary nature of local and global information for keyword expansion.

Limitations and Future Work

The authors acknowledge that while WANDER significantly improves performance, there is still a gap compared to fully supervised models. The paper suggests extending WANDER to other scenarios, such as multi-label classification, as a promising future direction. Additionally, while the paper demonstrates the effectiveness of the multi-stage approach, further exploration into optimizing the number of expansion iterations or the balance between local and global information could be beneficial.

Summary updated

Abstract

Scientific document classification is a critical task for a wide range of applications, but the cost of obtaining massive amounts of human-labeled data can be prohibitive. To address this challenge, we propose a weakly-supervised approach for scientific document classification using label names only. In scientific domains, label names often include domain-specific concepts that may not appear in the document corpus, making it difficult to match labels and documents precisely. To tackle this issue, we propose WANDER, which leverages dense retrieval to perform matching in the embedding space to capture the semantics of label names. We further design the label name expansion module to enrich the label name representations. Lastly, a self-training step is used to refine the predictions. The experiments on three datasets show that WANDER outperforms the best baseline by 11.9% on average. Our code will be published at https://github.com/ritaranx/wander.
BibTeX
@inproceedings{xu2023weakly,
  author = {Ran Xu and Yue Yu and Joyce C. Ho and Carl Yang},
  title = {{Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3592085},
}
Figure