TL;DR
This paper introduces WANDER, a novel weakly-supervised method for classifying scientific documents using only label names. WANDER overcomes the challenge of domain-specific label names not appearing in documents by employing dense retrieval to capture semantic similarity, expanding label names with relevant keywords, and refining predictions through self-training.
What Problem Does This Paper Address?
Classifying scientific documents into predefined categories is crucial for many applications, such as literature search and knowledge discovery. However, obtaining large amounts of human-labeled data for training supervised models is expensive and time-consuming. While weakly-supervised methods that use only label names as supervision offer a promising alternative, they struggle with scientific domains. This is because scientific label names are often highly specialized and may not directly appear in the documents, leading to a "coverage gap." Existing methods that rely on keyword matching or simple embedding similarity fail to bridge this gap effectively, resulting in significantly lower performance compared to fully supervised approaches on scientific datasets.
Methods
WANDER tackles this problem through a three-stage training process:
- Stage-I: Dense Retrieval with Label Names: Instead of relying on exact keyword matches, WANDER uses dense retrieval (DR) to find documents semantically similar to label names. This is achieved by encoding both label names (as queries) and documents into a shared embedding space. The DR model is first pre-trained using contrastive learning on the unlabeled corpus to improve its ability to align text representations. Then, label names are used as queries to retrieve a set of relevant documents from the unlabeled corpus. These retrieved documents, labeled according to their corresponding query label name, form an initial pseudo-labeled dataset.
- Stage-II: Expand Label Names with Local and Global Information: To address the limited semantic coverage of short label names, WANDER expands them. It extracts candidate keywords from the retrieved documents using a TF-IDF-like approach (local information) and then scores these keywords based on their semantic similarity to the label name using a pre-trained language model (global information). These local and global scores are combined using Reciprocal Rank Fusion (RRF) to rerank the candidate keywords. The top-ranked keyword is then appended to the label name. This expansion process is iterated multiple times, with the updated label names used to retrieve a new set of documents, progressively enriching the label representations and improving retrieval quality.
- Stage-III: Refine Classifier with Self-training: Finally, to leverage the entire unlabeled corpus and improve generalization, WANDER employs self-training. The classifier, fine-tuned on the pseudo-labeled data from Stage-II, is further trained on the unlabeled documents using its own high-confidence predictions as pseudo-labels. This iterative refinement process helps the model learn from a broader range of data.
Datasets
The paper evaluates WANDER on three scientific document classification datasets:
- MeSH: Biomedical domain.
- arXiv-Math: Mathematics domain.
- arXiv-CS: Computer Science domain.
These datasets are characterized by their domain-specific terminology and imbalanced label distributions, making them challenging for existing weakly-supervised methods.
Key Insights and Ideas
- Dense Retrieval is Key for Scientific Domains: The core insight is that dense retrieval, by capturing semantic similarity in an embedding space, is far more effective than traditional keyword matching for scientific documents where label names might not explicitly appear.
- Label Name Expansion is Crucial: Simply using raw label names is insufficient. Expanding them with relevant keywords, derived from both local document context and global semantic understanding, significantly improves the model's ability to capture class-specific information.
- Multi-stage Training Synergizes: Each stage of WANDER builds upon the previous one, creating a synergistic effect. Dense retrieval provides initial relevant data, label expansion refines the understanding of what's relevant, and self-training broadens the learning scope.
- Scientific Domains Pose Unique Challenges: The paper highlights that methods successful on general domains (like news or reviews) often fail on scientific data due to the specialized nature of terminology and the resulting coverage gap.
Key Results
WANDER consistently outperforms existing weakly-supervised baselines across all three datasets, achieving an average gain of 11.9% in Macro-F1 score. For instance, on the MeSH dataset, WANDER achieved a Macro-F1 of 81.9%, significantly outperforming the best baseline (FastClass) at 78.1%. The paper also shows that each stage of WANDER contributes to the final performance, with Stage-I and Stage-II alone already yielding substantial improvements. The ablation studies confirm the effectiveness of dense retrieval pre-training and the complementary nature of local and global information for keyword expansion.
Limitations and Future Work
The authors acknowledge that while WANDER significantly improves performance, there is still a gap compared to fully supervised models. The paper suggests extending WANDER to other scenarios, such as multi-label classification, as a promising future direction. Additionally, while the paper demonstrates the effectiveness of the multi-stage approach, further exploration into optimizing the number of expansion iterations or the balance between local and global information could be beneficial.
Summary updated