Bootstrapping Large-Scale Fine-Grained Contextual Advertising Classifier from Wikipedia

Yiping Jin, Vishakha Kadam, Dittaya WanvarieNAACL 2021

TL;DR

This paper introduces wiki2cat, a novel framework that leverages Wikipedia's category graph to automatically label a massive dataset of articles for fine-grained contextual advertising classification. This approach bypasses the need for manual labeling, enabling the creation of robust classifiers for thousands of categories, a significant challenge for traditional methods.

What Problem Does This Paper Address?

Contextual advertising aims to match ads with relevant content, but achieving this at a fine-grained level (e.g., "coupe" vs. "automotive") is difficult. Building supervised classifiers for such a large number of categories (thousands) requires vast amounts of labeled data, which is expensive and time-consuming to acquire. Existing methods often rely on manually curated rules or keywords, which are brittle and hard to maintain. Furthermore, fine-grained categories frequently overlap, making accurate classification a significant challenge. The paper addresses the gap of creating large-scale, fine-grained text classifiers without manual labeling by tapping into the rich, structured knowledge of Wikipedia.

Methods

The core of the wiki2cat framework involves three main steps:

  1. Mapping the Target Taxonomy to Wikipedia Categories: The paper first maps the target taxonomy (e.g., IAB taxonomy for advertising) to Wikipedia's extensive category graph. This involves lemmatizing category names, indexing them, and using string similarity to find closest matches. While a significant portion (82%) can be mapped automatically, a small manual curation step is used for the remaining categories.
  2. Labeling Wikipedia Articles by Category Graph Traversal: Once the mapping is established, the framework traverses the Wikipedia category graph. For each article, it identifies candidate categories by exploring paths from the mapped root categories. A competition-based algorithm, considering both shortest path distance and the number of unique paths between an article and a category, is used to assign the most relevant category labels. A pruning step is also employed to remove less relevant branches.
  3. Training Contextual Classifiers: The automatically labeled Wikipedia articles are then used to train standard classification models. The paper experiments with three models: a linear SVM with TF-IDF features, a centroid classifier, and a BERT-based model. The focus is on demonstrating the framework's ability to bootstrap classifiers, not on introducing novel model architectures.

Datasets

  • Training Data: The framework leverages Wikipedia dumps to generate a large, automatically labeled dataset. The paper reports labeling 1.16 million Wikipedia articles.
  • Evaluation Datasets:
    • Coarse-grained: Two datasets (nc-v2 and browsing) from Jin et al. (2020) covering 22 IAB tier-1 categories.
  • Fine-grained: A newly compiled dataset for fine-grained classification, comprising documents labeled with one of the IAB tier-2 categories. This dataset contains 332 fine-grained categories (out of 355) and 134k documents. The paper notes this is likely the only publicly available dataset for fine-grained contextual classification.

Key Insights and Ideas

  • Wikipedia as a Knowledge Source: The paper powerfully demonstrates that Wikipedia's structured category graph can be effectively mined to generate large-scale labeled datasets for text classification, bypassing the costly manual labeling process.
  • Graph Traversal for Labeling: The graph traversal approach, combined with a sophisticated weighting mechanism (considering path distance and number of unique paths), proves more effective than simpler methods like only considering immediate children or shortest paths.
  • Fine-grained Classification is Challenging: The results highlight that fine-grained classification is significantly harder than coarse-grained, especially when categories are semantically similar. Keyword voting performs poorly, indicating that category names alone are insufficient.
  • BERT's Limitations: While BERT is a powerful model, its performance was surprisingly lower than simpler models on news-related datasets due to sequence length limitations and potential domain mismatch between its pre-training data and the evaluation datasets.

Key Results

  • The wiki2cat framework, when used with SVM, achieved competitive performance on coarse-grained classification datasets, outperforming Dataless and Doc2vec baselines.
  • On the fine-grained classification dataset, wiki2cat (using SVM and centroid classifiers) significantly outperformed keyword voting and Dataless baselines, achieving 5-10% higher accuracy.
  • The graph labeling algorithm developed in the paper outperformed other graph-based labeling strategies, including those that did not use pruning.
  • The number of labeled training articles per fine-grained category varied, with the majority having between 100 to 10k articles.

Limitations and Future Work

  • Manual Curation: While significantly reduced, a small manual curation step was still required to map some IAB categories to Wikipedia categories.
  • BERT Performance: The paper acknowledges the limitations of BERT regarding sequence length and potential domain mismatch, suggesting further analysis and domain adaptation for BERT-based models.
  • Domain Adaptation: The authors are exploring the use of unlabeled in-domain documents for domain adaptation to improve classifier accuracy.
  • Category Similarity: The paper notes that the fine-grained classification task is challenging due to the semantic similarity of categories, suggesting that further research into handling such nuances is warranted.

Summary updated

BibTeX
@inproceedings{jin2021bootstrapping,
  author = {Yiping Jin and Vishakha Kadam and Dittaya Wanvarie},
  title = {{Bootstrapping Large-Scale Fine-Grained Contextual Advertising Classifier from Wikipedia}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure