Bootstrapping Domain-Specific Content Discovery on the Web

Kien Pham, Aécio S. R. Santos, Juliana FreireWWW 2019 · DOI · OpenReview

TL;DR

This paper introduces DISCO, a novel framework for automatically discovering relevant websites within a specific domain, even with a minimal set of initial seed websites. By mimicking human search behavior through iterative discovery, ranking, and selection of diverse search strategies, DISCO significantly outperforms existing methods in harvesting relevant content and achieving broad coverage.

What Problem Does This Paper Address?

Discovering relevant content on the web for specific domains is crucial for many applications, from combating human trafficking to analyzing financial markets. Traditional methods often rely on "focused crawlers" that require subject-matter experts (SMEs) to meticulously configure them. This involves manually searching for relevant websites, collecting representative pages to train a classifier, and providing seed pages for the crawler. This process is time-consuming and requires significant expertise. The core challenge is bootstrapping this discovery process efficiently, especially when SMEs have limited initial knowledge or resources to build accurate classifiers and seed crawlers.

Methods

DISCO tackles this challenge by proposing a domain-agnostic, iterative approach that doesn't require a pre-trained, accurate domain classifier. Its core components are:

  • Iterative Discovery and Ranking: DISCO starts with a small set of seed websites. In each iteration, it uses various discovery operators to find new web pages. These pages are then ranked based on their similarity to the seed set, approximating how a human would identify relevant content. The top-ranked pages are then used as new seeds for the next iteration.
  • Multiple Discovery Operators: DISCO integrates four distinct discovery strategies:
  • Keyword Search: Leverages search engine APIs with keywords derived from seed websites.
  • Related Search: Uses search engine APIs to find sites similar to a given URL.
  • Forward Crawling: Explores links found on discovered web pages.
  • Backward Crawling: Utilizes backlink search to find pages pointing to known relevant sites.
  • Multi-Armed Bandit (MAB) for Operator Selection: To dynamically choose the most effective discovery operator at each step, DISCO employs a UCB1 MAB algorithm. This balances exploration (trying different operators) and exploitation (using the most successful ones), adapting to the domain's characteristics.
  • Ensemble Ranking Function: Recognizing that no single ranking method is universally optimal, DISCO proposes an ensemble approach that combines the results of multiple ranking functions (e.g., Jaccard, Cosine similarity, Bayesian Sets, One-Class SVM, Binomial Regression). This ensemble consistently achieves higher precision and robustness.

Datasets

The experiments were conducted using real-world data from four "social-good" domains:

  • Human Trafficking (HT): Classified ads and escort-related websites.
  • Weapon Forums (Forum): Discussion forums about firearms and weapons.
  • Weapon Marketplaces (Market): Websites for buying and selling firearms and weapons.
  • Stock Promotions (SEC): Websites posting stock promotion alerts, relevant for micro-cap fraud investigations.

For training and evaluation, SMEs provided:

  • Seed Websites: A small initial set of relevant websites for each domain.
  • Training Examples: Larger sets of manually labeled relevant and non-relevant websites to create ground-truth classifiers for evaluation purposes.

Key Insights and Ideas

  • Classifier-Free Bootstrapping: The most significant insight is that effective domain discovery can be achieved without relying on a pre-existing, accurate domain classifier. This dramatically lowers the barrier to entry for SMEs.
  • Mimicking Human Search: The iterative discovery-ranking-selection loop closely mirrors how humans explore information online, making the process more intuitive and effective.
  • Adaptive Operator Selection is Crucial: Different discovery operators perform vastly differently across domains. A dynamic, adaptive selection mechanism (like MAB) is essential for maximizing harvest rates and coverage, rather than relying on a fixed strategy.
  • Ensemble Ranking for Robustness: Combining multiple ranking signals leads to more reliable and accurate ranking of discovered pages, which is critical for feeding high-quality candidates back into the discovery process.
  • Social-Good Domains as a Testbed: Using these challenging, often less structured domains highlights the robustness and applicability of DISCO beyond typical web search scenarios.

Key Results

DISCO, particularly with the BANDIT configuration (using MAB for operator selection), demonstrated significant improvements:

  • Harvest Rate: Achieved at least 300% higher harvest rate compared to state-of-the-art baselines across the evaluated domains.
  • Coverage: Consistently attained the largest intersection of discovered relevant websites, suggesting it explores a broader portion of the web.
  • Ranking Performance: The ENSEMBLE ranking function outperformed individual ranking methods in P@k, mean rank, and median rank metrics across domains.
  • Seed Website Sensitivity: Experiments showed that precision improves with more seed websites, but plateaus around 15 seeds, indicating that DISCO is effective even with a small initial set.

Limitations and Future Work

  • Domain Representativeness: The current ranking approach is most effective for domains where n-gram features are informative and the provided seed websites are truly representative of the domain.
  • Keyword Sensitivity: Keyword search, a component of DISCO, can be sensitive to the quality of initial seed keywords, which can be challenging to select for some domains.

Future work suggested by the authors includes:

  • Image-Based Search: Exploring image-based search for domains rich in visual content, such as weapon ads and human trafficking sites.
  • Leveraging Advanced Text Representations: Investigating newer techniques for text representation and document distance computation to potentially further improve ranking performance.

Summary updated

Abstract

The ability to continuously discover domain-specific content from the Web is critical for many applications. While focused crawling strategies have been shown to be effective for discovery, configuring a focused crawler is difficult and time-consuming. Given a domain of interest $D$, subject-matter experts (SMEs) must search for relevant websites and collect a set of representative Web pages to serve as training examples for creating a classifier that recognizes pages in $D$, as well as a set of pages to seed the crawl. In this paper, we propose DISCO, an approach designed to bootstrap domain-specific search. Given a small set of websites, DISCO aims to discover a large collection of relevant websites. DISCO uses a ranking-based framework that mimics the way users search for information on the Web: it iteratively discovers new pages, distills, and ranks them. It also applies multiple discovery strategies, including keyword-based and related queries issued to search engines, backward and forward crawling. By systematically combining these strategies, DISCO is able to attain high harvest rates and coverage for a variety of domains. We perform extensive experiments in four social-good domains, using data gathered by SMEs in the respective domains, and show that our approach is effective and outperforms state-of-the-art methods.
BibTeX
@inproceedings{pham2019bootstrapping,
  author = {Kien Pham and Aécio S. R. Santos and Juliana Freire},
  title = {{Bootstrapping Domain-Specific Content Discovery on the Web}},
  booktitle = {The Web Conference},
  year = {2019},
  doi = {10.1145/3308558.3313709},
}
Figure