TL;DR
MotifClass is a new framework for text classification that works without human-annotated labels by leveraging document metadata (like authors, venues, or years). By treating metadata combinations as "motifs" and modeling their specificity, the system automatically generates high-quality pseudo-labeled training data to train a robust classifier.
What Problem Does This Paper Address?
In many real-world scenarios, we want to classify documents (like research papers or product reviews) but lack the budget or time for manual labeling. "Weakly supervised" methods attempt to solve this using only category names (e.g., "Natural Language Processing") as a starting point.
However, most existing methods focus purely on the text. They ignore the rich metadata that often accompanies documents. Even when metadata is used, it is typically treated as "flat" information. The authors argue that two things are missing:
- Higher-order interactions: A specific author publishing at a specific venue in a specific year is a much stronger signal of a category than any of those facts alone.
- Metadata Specificity: Some metadata is too broad. For example, the term "language" is too vague to identify an NLP paper, whereas "EMNLP" (a specific conference) is highly indicative.
Methods
The authors propose MotifClass, which operates in three main stages:
- Indicative Motif Selection: The system models documents and metadata as a Heterogeneous Information Network (HIN). It defines "motifs"—subgraph patterns like Venue-Year or Author-Author. It then uses a joint representation learning process to embed these motifs and category names into the same space. Crucially, they use the von Mises-Fisher (vMF) distribution, where a "concentration parameter" ($\kappa$) represents the specificity of a motif. This allows the system to filter out metadata that is too generic.
- Pseudo-labeled Data Collection: To create a training set, MotifClass uses two complementary strategies:
- Retrieval: It finds existing unlabeled documents that contain highly indicative motifs for a category.
- Generation: It uses the learned embeddings to "synthesize" new document representations that are semantically aligned with the target categories.
- Classifier Training: Finally, a standard text classifier (in this case, a Kim-CNN) is trained on this automatically generated "pseudo-labeled" dataset.
Datasets
The framework was evaluated on two large-scale, real-world datasets:
- MAG-CS: A collection of ~203,000 academic papers from the Microsoft Academic Graph across 20 computer science categories. Metadata includes Authors, Venues, and Years.
- Amazon: A dataset of 100,000 product reviews across 10 categories. Metadata includes User IDs and Product IDs.
Key Insights and Ideas
- Specificity is a Mathematical Feature: By using the vMF distribution, the model doesn't just learn what a word means, but how narrow its meaning is. This prevents the model from being misled by common but non-descriptive metadata (like a very famous author who writes in many different fields).
- Motifs Capture Context: The paper demonstrates that nearly half of the most useful signals for classification come from higher-order motifs (combinations) rather than single metadata fields. For example, "KDD" (a conference) combined with "Jiawei Han" (an author) is a near-certain indicator of the "Data Mining" category.
- Retrieval vs. Generation: The authors found that retrieval provides high-quality linguistic patterns, while generation ensures the model has enough volume and diversity of data to generalize well.
Key Results
MotifClass consistently outperformed state-of-the-art weakly supervised baselines (like WeSTClass and LOTClass) and HIN-based embedding methods:
- Performance Boost: On the MAG-CS dataset, MotifClass achieved a Micro-F1 score of 0.571, significantly higher than the best metadata-aware baseline (0.525).
- Ablation Success: Removing either the "higher-order" motif information or the "specificity" modeling led to a measurable drop in accuracy, proving both components are vital.
- Efficiency: Despite dealing with complex network structures, MotifClass is remarkably fast—running 26.6 times faster than some iterative motif-based baselines on the MAG-CS dataset.
Limitations and Future Work
The authors acknowledge a few areas for growth:
- Single-Label Constraint: The current model assumes each document belongs to exactly one category. Real-world data is often multi-label or hierarchical.
- Manual Motif Patterns: While the system selects specific instances automatically, the types of motifs (e.g., "look for Author-Venue combos") are currently specified by the user. The authors suggest that automatically detecting these patterns under weak supervision is a promising next step.
- Language Models: While they tested BERT, it struggled with the "fixed vocabulary" problem when dealing with unique metadata like Product IDs. Future work could involve better integrating large language models with open-vocabulary metadata.
Summary updated