Abstract
Dheeraj Mekala, Xinyang Zhang, Jingbo Shang. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
In the field of Natural Language Processing (NLP), labeling massive datasets remains a significant bottleneck. Weakly supervised learning attempts to mitigate this by allowing users to provide a few "seed words" to train a classifier. However, most existing methods ignore a goldmine of information: metadata. Whether it is a research paper's author, a news source, or a timestamp, metadata often contains powerful signals. For example, an author like "Geoffrey Hinton" is a much stronger indicator of a "Machine Learning" category than many common keywords.
META (Metadata-Empowered Weak Supervision) is a novel framework that leverages metadata as a source of weak supervision. By modeling text and metadata as a "text-rich network" and using network motifs to capture complex relationships, META automatically discovers highly indicative "seed motifs" to train high-quality classifiers with minimal human effort.
Methodology
The META framework operates through an iterative, three-step bootstrapping process designed to systematically distill reliable supervision from messy, multi-typed metadata.
- Text-Rich Network Construction: The entire corpus is represented as a heterogeneous network where documents, words, and metadata entities (authors, years, etc.) are nodes, and their associations are edges.
- Network Motifs: To capture complex relationships, META uses "motifs"—small subgraph patterns. Instead of just looking at an author, META might look at an Author-Year motif. This is crucial because an author’s research interests might shift over time; a specific author-year combination can be a much "purer" signal for a category than the author alone.
- The Iterative Loop:
- Pseudo-Labeling: Using current seed words and motifs, the system assigns temporary labels to unlabeled documents based on a count-based technique.
- Classifier Training: A Hierarchical Attention Network (HAN) is trained on these pseudo-labeled documents.
- Unified Ranking & Expansion: META uses the trained classifier’s predictions to run a Personalized PageRank (PPR) algorithm across the network. It ranks words and motif instances based on how "relevant" and "exclusive" they are to a specific class. Top-ranked items are added to the seed set, and the loop repeats.
Experimental Evaluation
The authors evaluated META on two real-world datasets: DBLP (computer science papers with Author and Year metadata) and Book Graph (Goodreads book descriptions with Author, Publisher, and Year metadata).
Key Results
META consistently outperformed state-of-the-art weakly supervised baselines that rely only on text (like WeSTClass) or simpler metadata treatments. On the Book Graph dataset, META achieved a Micro-F1 score of 0.62, a massive jump from the 0.42 achieved by the text-only WeSTClass. Ablation studies confirmed that removing the metadata expansion (META-NoMeta) led to a noticeable drop in performance, proving that motif-based supervision provides signals that text alone cannot capture. The system is efficient, typically reaching peak performance within 6 to 7 iterations.
Discussion and Insights
- Contextualizing Metadata: While a single piece of metadata (like a year) might be weak on its own, combining it with another (like an author) creates a highly specific and accurate label indicator.
- Exclusiveness is Key: A good seed isn't just frequent in its class; it must be rare in others. The ranking mechanism specifically penalizes "flat" distributions where a motif appears across many categories.
- Adaptive Growth: Unlike methods that add a fixed number of seeds per round, META’s expansion is data-driven. If a category has many clear indicators, it grows faster; if the signals are noisy, it stays cautious.
Limitations and Future Work
The authors acknowledge that META currently focuses on finding metadata that indicates a class. Future versions could look for "negative indicators"—metadata that suggests a document definitely does not belong to a certain category. Additionally, integrating other forms of weak supervision, such as heuristic rules, could make the system even more robust.
Reference: Mekala, D., Zhang, X., & Shang, J. (2020). META: Metadata-Empowered Weak Supervision for Text Classification. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 8351–8361.
Summary updated