TL;DR
This paper introduces DISCO, a novel framework for automatically discovering relevant websites within a specific domain, even with a minimal set of initial seed websites. By mimicking human search behavior through iterative discovery, ranking, and selection of diverse search strategies, DISCO significantly outperforms existing methods in harvesting relevant content and achieving broad coverage.
What Problem Does This Paper Address?
Discovering relevant content on the web for specific domains is crucial for many applications, from combating human trafficking to analyzing financial markets. Traditional methods often rely on "focused crawlers" that require subject-matter experts (SMEs) to meticulously configure them. This involves manually searching for relevant websites, collecting representative pages to train a classifier, and providing seed pages for the crawler. This process is time-consuming and requires significant expertise. The core challenge is bootstrapping this discovery process efficiently, especially when SMEs have limited initial knowledge or resources to build accurate classifiers and seed crawlers.
Methods
DISCO tackles this challenge by proposing a domain-agnostic, iterative approach that doesn't require a pre-trained, accurate domain classifier. Its core components are:
- Iterative Discovery and Ranking: DISCO starts with a small set of seed websites. In each iteration, it uses various discovery operators to find new web pages. These pages are then ranked based on their similarity to the seed set, approximating how a human would identify relevant content. The top-ranked pages are then used as new seeds for the next iteration.
- Multiple Discovery Operators: DISCO integrates four distinct discovery strategies:
- Keyword Search: Leverages search engine APIs with keywords derived from seed websites.
- Related Search: Uses search engine APIs to find sites similar to a given URL.
- Forward Crawling: Explores links found on discovered web pages.
- Backward Crawling: Utilizes backlink search to find pages pointing to known relevant sites.
- Multi-Armed Bandit (MAB) for Operator Selection: To dynamically choose the most effective discovery operator at each step, DISCO employs a UCB1 MAB algorithm. This balances exploration (trying different operators) and exploitation (using the most successful ones), adapting to the domain's characteristics.
- Ensemble Ranking Function: Recognizing that no single ranking method is universally optimal, DISCO proposes an ensemble approach that combines the results of multiple ranking functions (e.g., Jaccard, Cosine similarity, Bayesian Sets, One-Class SVM, Binomial Regression). This ensemble consistently achieves higher precision and robustness.
Datasets
The experiments were conducted using real-world data from four "social-good" domains:
- Human Trafficking (HT): Classified ads and escort-related websites.
- Weapon Forums (Forum): Discussion forums about firearms and weapons.
- Weapon Marketplaces (Market): Websites for buying and selling firearms and weapons.
- Stock Promotions (SEC): Websites posting stock promotion alerts, relevant for micro-cap fraud investigations.
For training and evaluation, SMEs provided:
- Seed Websites: A small initial set of relevant websites for each domain.
- Training Examples: Larger sets of manually labeled relevant and non-relevant websites to create ground-truth classifiers for evaluation purposes.
Key Insights and Ideas
- Classifier-Free Bootstrapping: The most significant insight is that effective domain discovery can be achieved without relying on a pre-existing, accurate domain classifier. This dramatically lowers the barrier to entry for SMEs.
- Mimicking Human Search: The iterative discovery-ranking-selection loop closely mirrors how humans explore information online, making the process more intuitive and effective.
- Adaptive Operator Selection is Crucial: Different discovery operators perform vastly differently across domains. A dynamic, adaptive selection mechanism (like MAB) is essential for maximizing harvest rates and coverage, rather than relying on a fixed strategy.
- Ensemble Ranking for Robustness: Combining multiple ranking signals leads to more reliable and accurate ranking of discovered pages, which is critical for feeding high-quality candidates back into the discovery process.
- Social-Good Domains as a Testbed: Using these challenging, often less structured domains highlights the robustness and applicability of DISCO beyond typical web search scenarios.
Key Results
DISCO, particularly with the BANDIT configuration (using MAB for operator selection), demonstrated significant improvements:
- Harvest Rate: Achieved at least 300% higher harvest rate compared to state-of-the-art baselines across the evaluated domains.
- Coverage: Consistently attained the largest intersection of discovered relevant websites, suggesting it explores a broader portion of the web.
- Ranking Performance: The ENSEMBLE ranking function outperformed individual ranking methods in P@k, mean rank, and median rank metrics across domains.
- Seed Website Sensitivity: Experiments showed that precision improves with more seed websites, but plateaus around 15 seeds, indicating that DISCO is effective even with a small initial set.
Limitations and Future Work
- Domain Representativeness: The current ranking approach is most effective for domains where n-gram features are informative and the provided seed websites are truly representative of the domain.
- Keyword Sensitivity: Keyword search, a component of DISCO, can be sensitive to the quality of initial seed keywords, which can be challenging to select for some domains.
Future work suggested by the authors includes:
- Image-Based Search: Exploring image-based search for domains rich in visual content, such as weapon ads and human trafficking sites.
- Leveraging Advanced Text Representations: Investigating newer techniques for text representation and document distance computation to potentially further improve ranking performance.
Summary updated