MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Information

Yu Zhang, Shweta Garg, Yu Meng, Xiusi Chen, Jiawei Han

Abstract

Weakly supervised text classification aims to categorize documents using only category surface names, bypassing the need for expensive manual annotations. While most existing methods rely solely on textual content, many documents are accompanied by rich metadata—such as authors, venues, and publication years—that can serve as powerful category indicators. We introduce MotifClass, a framework that models these relationships through a Heterogeneous Information Network (HIN) and captures higher-order structures using "motifs." By selecting category-indicative motif instances and generating pseudo-labeled training data, MotifClass significantly outperforms state-of-the-art weakly supervised baselines.

Introduction

In the era of big data, the ability to classify vast document collections without human intervention is invaluable. Traditional supervised learning requires large labeled datasets, which are often unavailable or costly to produce. Weakly supervised methods offer a solution by using minimal input, such as category names (e.g., "Natural Language Processing" or "Data Mining").

However, a significant oversight in current research is the neglect of document metadata. In academic publishing, for instance, the combination of a specific author and a specific conference is often a much stronger signal of a paper's topic than any single word in its title. MotifClass addresses this by treating documents and their metadata as nodes in a Heterogeneous Information Network (HIN).

A Heterogeneous Information Network (HIN) representing documents and their associated metadata.

The Challenge: Specificity and Higher-Order Interactions

Two main challenges arise when using metadata for classification:

  1. Higher-order interactions: Single metadata fields (e.g., a year) are often weak indicators. However, combinations (e.g., Author + Venue) provide much higher precision.
  2. Metadata Specificity: Not all metadata is created equal. A term like "language" is too broad to uniquely identify an NLP paper, whereas a venue like "EMNLP" is highly specific.

MotifClass solves these by defining "motifs"—subgraph patterns that represent specific combinations of metadata—and evaluating their specificity.

The MotifClass Framework

The framework consists of three primary stages:

1. Indicative Motif Selection

MotifClass identifies which metadata combinations (motif instances) are most representative of a category. It uses a joint representation learning approach, embedding both motifs and category names into a shared space. To model specificity, the authors employ the von Mises-Fisher (vMF) distribution. In this model, the "concentration parameter" ($\kappa$) acts as a proxy for specificity: a high $\kappa$ indicates a motif instance that is tightly associated with a specific category, while a low $\kappa$ suggests a generic or "broad" motif.

Comparison of category-indicative motif instances versus those that are too broad or dissimilar.

2. Pseudo-labeled Data Collection

Once indicative motifs are identified, the system builds a training set through two methods:

  • Retrieval: It identifies unlabeled documents in the corpus that contain these highly indicative motif instances.
  • Generation: It synthesizes new document representations that are semantically aligned with the target categories, ensuring a diverse and voluminous training set.

3. Classifier Training

Finally, a standard text classifier (such as a Kim-CNN) is trained on this pseudo-labeled data. This allows the model to learn the linguistic patterns associated with the categories, guided by the high-quality labels derived from metadata motifs.

Experimental Results

The authors evaluated MotifClass on two large-scale datasets: MAG-CS (computer science papers) and Amazon (product reviews).

  • Superior Accuracy: On the MAG-CS dataset, MotifClass achieved a Micro-F1 score of 0.571, outperforming the best metadata-aware baseline (0.525) and significantly beating text-only methods like WeSTClass.
  • The Power of Higher-Order Data: Ablation studies revealed that nearly 50% of the most useful signals came from higher-order motifs rather than individual metadata fields.
  • Efficiency: MotifClass is highly scalable. It processed the MAG-CS dataset 26.6 times faster than previous iterative motif-based approaches.

Conclusion

MotifClass demonstrates that metadata is not just "extra" information but a core component of document identity. By mathematically modeling the specificity of metadata combinations, MotifClass provides a robust, efficient, and highly accurate framework for text classification in the absence of human labels. Future work will explore extending this to multi-label and hierarchical classification tasks.

Summary updated

Abstract

We study the problem of weakly supervised text classification, which aims to classify text documents into a set of pre-defined categories with category surface names only and without any annotated training document provided. Most existing classifiers leverage textual information in each document. However, in many domains, documents are accompanied by various types of metadata (e.g., authors, venue, and year of a research paper). These metadata and their combinations may serve as strong category indicators in addition to textual contents. In this paper, we explore the potential of using metadata to help weakly supervised text classification. To be specific, we model the relationships between documents and metadata via a heterogeneous information network. To effectively capture higher-order structures in the network, we use motifs to describe metadata combinations. We propose a novel framework, named MotifClass, which (1) selects category-indicative motif instances, (2) retrieves and generates pseudo-labeled training samples based on category names and indicative motif instances, and (3) trains a text classifier using the pseudo training data. Extensive experiments on real-world datasets demonstrate the superior performance of MotifClass to existing weakly supervised text classification approaches. Further analysis shows the benefit of considering higher-order metadata information in our framework.
BibTeX
@inproceedings{zhang2022motifclass,
  author = {Yu Zhang and Shweta Garg and Yu Meng and Xiusi Chen and Jiawei Han},
  title = {{MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Information}},
  booktitle = {WSDM},
  year = {2022},
  doi = {10.1145/3488560.3498384},
}

Loading related content...

Figure