MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Information

Yu Zhang, Shweta Garg, Yu Meng, Xiusi Chen, Jiawei Han

TL;DR

MotifClass is a new framework for text classification that works without human-annotated labels by leveraging document metadata (like authors, venues, or years). By treating metadata combinations as "motifs" and modeling their specificity, the system automatically generates high-quality pseudo-labeled training data to train a robust classifier.

What Problem Does This Paper Address?

In many real-world scenarios, we want to classify documents (like research papers or product reviews) but lack the budget or time for manual labeling. "Weakly supervised" methods attempt to solve this using only category names (e.g., "Natural Language Processing") as a starting point.

However, most existing methods focus purely on the text. They ignore the rich metadata that often accompanies documents. Even when metadata is used, it is typically treated as "flat" information. The authors argue that two things are missing:

  1. Higher-order interactions: A specific author publishing at a specific venue in a specific year is a much stronger signal of a category than any of those facts alone.
  2. Metadata Specificity: Some metadata is too broad. For example, the term "language" is too vague to identify an NLP paper, whereas "EMNLP" (a specific conference) is highly indicative.

Methods

The authors propose MotifClass, which operates in three main stages:

  1. Indicative Motif Selection: The system models documents and metadata as a Heterogeneous Information Network (HIN). It defines "motifs"—subgraph patterns like Venue-Year or Author-Author. It then uses a joint representation learning process to embed these motifs and category names into the same space. Crucially, they use the von Mises-Fisher (vMF) distribution, where a "concentration parameter" ($\kappa$) represents the specificity of a motif. This allows the system to filter out metadata that is too generic.
  2. Pseudo-labeled Data Collection: To create a training set, MotifClass uses two complementary strategies:
  3. Retrieval: It finds existing unlabeled documents that contain highly indicative motifs for a category.
  4. Generation: It uses the learned embeddings to "synthesize" new document representations that are semantically aligned with the target categories.
  5. Classifier Training: Finally, a standard text classifier (in this case, a Kim-CNN) is trained on this automatically generated "pseudo-labeled" dataset.

Datasets

The framework was evaluated on two large-scale, real-world datasets:

  • MAG-CS: A collection of ~203,000 academic papers from the Microsoft Academic Graph across 20 computer science categories. Metadata includes Authors, Venues, and Years.
  • Amazon: A dataset of 100,000 product reviews across 10 categories. Metadata includes User IDs and Product IDs.

Key Insights and Ideas

  • Specificity is a Mathematical Feature: By using the vMF distribution, the model doesn't just learn what a word means, but how narrow its meaning is. This prevents the model from being misled by common but non-descriptive metadata (like a very famous author who writes in many different fields).
  • Motifs Capture Context: The paper demonstrates that nearly half of the most useful signals for classification come from higher-order motifs (combinations) rather than single metadata fields. For example, "KDD" (a conference) combined with "Jiawei Han" (an author) is a near-certain indicator of the "Data Mining" category.
  • Retrieval vs. Generation: The authors found that retrieval provides high-quality linguistic patterns, while generation ensures the model has enough volume and diversity of data to generalize well.

Key Results

MotifClass consistently outperformed state-of-the-art weakly supervised baselines (like WeSTClass and LOTClass) and HIN-based embedding methods:

  • Performance Boost: On the MAG-CS dataset, MotifClass achieved a Micro-F1 score of 0.571, significantly higher than the best metadata-aware baseline (0.525).
  • Ablation Success: Removing either the "higher-order" motif information or the "specificity" modeling led to a measurable drop in accuracy, proving both components are vital.
  • Efficiency: Despite dealing with complex network structures, MotifClass is remarkably fast—running 26.6 times faster than some iterative motif-based baselines on the MAG-CS dataset.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Single-Label Constraint: The current model assumes each document belongs to exactly one category. Real-world data is often multi-label or hierarchical.
  • Manual Motif Patterns: While the system selects specific instances automatically, the types of motifs (e.g., "look for Author-Venue combos") are currently specified by the user. The authors suggest that automatically detecting these patterns under weak supervision is a promising next step.
  • Language Models: While they tested BERT, it struggled with the "fixed vocabulary" problem when dealing with unique metadata like Product IDs. Future work could involve better integrating large language models with open-vocabulary metadata.

Summary updated

Abstract

We study the problem of weakly supervised text classification, which aims to classify text documents into a set of pre-defined categories with category surface names only and without any annotated training document provided. Most existing classifiers leverage textual information in each document. However, in many domains, documents are accompanied by various types of metadata (e.g., authors, venue, and year of a research paper). These metadata and their combinations may serve as strong category indicators in addition to textual contents. In this paper, we explore the potential of using metadata to help weakly supervised text classification. To be specific, we model the relationships between documents and metadata via a heterogeneous information network. To effectively capture higher-order structures in the network, we use motifs to describe metadata combinations. We propose a novel framework, named MotifClass , which (1) selects category-indicative motif instances, (2) retrieves and generates pseudo-labeled training samples based on category names and indicative motif instances, and (3) trains a text classifier using the pseudo training data. Extensive experiments on real-world datasets demonstrate the superior performance of MotifClass to existing weakly supervised text classification approaches. Further analysis shows the benefit of considering higher-order metadata information in our framework.
BibTeX
@inproceedings{zhang2022motifclass,
  author = {Yu Zhang and Shweta Garg and Yu Meng and Xiusi Chen and Jiawei Han},
  title = {{MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Information}},
  booktitle = {WSDM},
  year = {2022},
  doi = {10.1145/3488560.3498384},
}

Loading related content...

Figure