A reference collection for web spam

Carlos Castillo, Debora Donato, Luca Becchetti, Paolo Boldi, Stefano Leonardi, Massimo Santini, Sebastiano VignaSIGIR 2006 · DOI

TL;DR

This paper introduces the WEBSPAM-UK2006 collection, a novel, publicly available dataset of manually annotated web pages designed to advance research in web spam detection. By providing a large, diverse, and richly labeled dataset, it aims to enable more robust development and comparison of spam detection algorithms.

What Problem Does This Paper Address?

The paper tackles the significant challenge of accurately detecting and mitigating web spam. Web spam refers to deceptive practices used to manipulate search engine rankings, undermining the credibility of search results and user trust. A major hurdle in this research area has been the lack of standardized, publicly available datasets for training and evaluating spam detection models. This forces researchers to create their own datasets, which are often biased, small, or not comparable, hindering progress and making it difficult to benchmark new techniques. The authors highlight the "arms race" between spammers and search engines, emphasizing the need for robust adversarial information retrieval techniques, which in turn require good data.

Methods

The authors describe a multi-phase process for creating their reference collection:

  • Web Crawling: They used the UbiCrawler to collect a large dataset of UK web pages, focusing on the .uk domain. The crawl was conducted in a breadth-first search manner for cross-host links and depth-first for local links, with specific depth and page limits per host to manage scale. The collected data was stored in the WARC/0.9 format.
  • Elaboration of Web Spam Guidelines and Classification Interface: They reviewed existing literature and guidelines from major search engines (Google, Yahoo, MSN) to create a set of criteria for identifying spam. A web-based interface was developed to facilitate the classification process, presenting reviewers with a "work unit" of hosts to classify, along with host information (in/out-links, WHOIS) and sample pages.
  • Labelling: A large group of volunteers (33 in total) from relevant mailing lists were recruited to classify hosts. Each host was assigned to at least two judges, with a third judge brought in for cases with conflicting evaluations. The volunteers were given guidelines and access to the classification interface.
  • Post-processing: Judgments were anonymized, and special "judges" were added to incorporate existing knowledge: one for hosts listed in the Open Directory Project (ODP) and another for specific .uk domains (e.g., .ac.uk, .gov.uk) that were not manually reviewed.

Datasets

The primary dataset introduced is the WEBSPAM-UK2006 collection. This collection consists of:

  • Web Pages: Approximately 77.9 million pages from over 11,000 hosts within the .uk domain.
  • Web Graph: A compressed web graph representing links between pages.
  • Labels: Manually annotated labels for hosts, classifying them as "normal," "spam," or "borderline." The dataset includes 6,552 evaluations from 33 human judges.

The authors also reference and discuss several existing datasets used in prior web spam literature, providing a comparative table (Table 1) that highlights their limitations (e.g., bias, lack of availability).

Key Insights and Ideas

  • Subjectivity of Spam: The evaluation process revealed that while there's a general consensus on clear-cut spam and non-spam cases, a significant "gray area" exists for borderline cases. This subjectivity is inherent in human judgment and poses a challenge for automated detection.
  • Difficulty in Defining Spam: The paper echoes the sentiment that defining web spam precisely is difficult, often relying on the "I know it when I see it" principle. This highlights the need for clear, actionable guidelines for both human annotators and automated systems.
  • Agreement Metrics are Crucial: The study emphasizes the importance of measuring inter-rater reliability (using Kappa statistics and agreement indices) to understand the consistency of human judgments. This is vital for building trust in the dataset and for evaluating the performance of spam detection models.
  • Prevalence of Spam Aspects: The qualitative analysis of spam hosts identified common characteristics like keywords in URLs and anchor text, multiple sponsored links, and external ad units, providing insights into typical spamming techniques.

Key Results

  • Dataset Size: The WEBSPAM-UK2006 collection is substantial, comprising 77.9 million pages and over 3x10^9 edges, from 11,000 hosts.
  • Label Distribution: Out of 6,552 evaluations, the most common label was "normal" (61.75%), followed by "spam" (22.08%), "borderline" (10.82%), and "can not classify" (5.34%).
  • Spamicity Metric: After applying a spamicity metric (based on weighted judgments), 71% of hosts were classified as normal, 25% as spam, and 4% as undecided.
  • Inter-Rater Reliability: The overall Kappa statistic for agreement among judges was 0.56 (moderate agreement). However, when restricting to "normal" and "spam" categories only, the agreement rose to 0.82 (almost perfect). Agreement on "borderline" cases was significantly lower (0.11, slight agreement).
  • Agreement Index: The average agreement index was around 75-83%, indicating a reasonable but not perfect consensus among reviewers, even with higher overlap in reviewed pages.

Limitations and Future Work

  • Subjectivity of Borderline Cases: The authors acknowledge the difficulty in evaluating borderline cases, which leads to lower agreement scores for this category. This suggests a need for more refined guidelines or potentially different classification schemes for such cases.
  • Limited Scope: The collection is primarily focused on .uk domain hosts. While this provides a specific context, future work could expand to a more global or diverse set of domains.
  • Need for Stricter Definitions: The observed partial consensus on what constitutes spam suggests a need for stricter and more unambiguous definitions of web spam to improve consistency in labeling and detection.
  • Future Collections: The authors propose that future collections should incorporate a "checklist" of spam aspects to guide reviewers and improve objectivity.
  • Staying Ahead of Spammers: The paper notes that web spam is a dynamic and challenging area, and future research should focus on developing general methods that can adapt to new spamming techniques.

Summary updated

BibTeX
@article{castillo2006reference,
  author = {Carlos Castillo and Debora Donato and Luca Becchetti and Paolo Boldi and Stefano Leonardi and Massimo Santini and Sebastiano Vigna},
  title = {{A reference collection for web spam}},
  journal = {SIGIR},
  year = {2006},
  doi = {10.1145/1189702.1189703},
}
Figure