Syskill and Webert: Identifying Interesting Web Sites

AAAI 1996

TL;DR

This paper introduces Syskill & Webert, an intelligent agent that learns individual user preferences to identify interesting web pages. By analyzing user ratings of web content, it builds personalized profiles that can suggest relevant links or filter search results, significantly improving the efficiency of navigating the vast World Wide Web.

What Problem Does This Paper Address?

The World Wide Web is an ever-expanding repository of information, making it increasingly difficult for users to find content relevant to their specific interests. Traditional search engines provide a broad range of results, often requiring users to sift through numerous irrelevant pages. This paper tackles the challenge of personalizing web navigation by developing an agent that learns what a specific user finds interesting, thereby reducing the time and effort needed to discover valuable information.

Methods

Syskill & Webert employs a machine learning approach to build user profiles. The core idea is to have users rate web pages they visit (e.g., "hot," "lukewarm," or "cold"). The agent then analyzes the HTML content of these rated pages to learn a profile for that user's topic of interest.

Key methodological aspects include:

  • User Profiling: Instead of a single profile for a user, Syskill & Webert learns a separate profile for each topic a user is interested in. This allows for more nuanced and accurate learning, as interests in different topics are unlikely to be driven by the same factors.
  • Feature Extraction: HTML documents are converted into feature vectors. Initially, this involved using words from the HTML source. The paper explores using "informative words" identified through information-based approaches (similar to decision tree learning) to represent pages.
  • Learning Algorithms: The paper compares several machine learning algorithms for profile learning, including Naive Bayesian classifiers, nearest neighbor, ID3, perceptrons, and multi-layer networks.
  • Two Modes of Operation:
    1. Link Suggestion: The agent annotates links on an "index page" (a manually curated page of links on a specific topic) with predictions of user interest.
  • Search Filtering: The agent constructs queries for search engines like LYCOS using the learned user profile and can then filter the search results.
  • Efficiency Optimization: To reduce network overhead, the system can analyze only an initial segment of a web page (e.g., the first 512-2048 characters) for learning and prediction, often achieving comparable accuracy to analyzing the entire document.

Datasets

The paper describes experiments conducted using manually collected "index pages" covering various topics. These topics and their associated index page URLs are listed in Table 2:

  • Biomedical: http://golgi.harvard.edu/biopages/medicine.html
  • Lycos: (No specific URL provided, likely referring to general LYCOS searches)
  • Bands (listening): http://www.iuma.com/IUMA-2.0/olas/location/USA.html
  • Bands (reading): http://www.iuma.com/IUMA-2.0/olas/location/USA.html
  • Movies: http://rte66.com/Movies/blurb.html
  • Protein: http://golgi.harvard.edu/sequences.html

The experiments involved users rating pages within these topics, creating datasets for training and evaluating the learning algorithms.

Key Insights and Ideas

  • Personalization is Key: The fundamental insight is that individual user preferences are highly specific and can be learned, leading to a more efficient web experience than generic approaches.
  • Topic-Specific Profiles: Learning separate profiles for different user interests is more effective than a single, monolithic profile.
  • Early Segments Suffice: Surprisingly, analyzing only the initial portion of a web page can yield nearly the same prediction accuracy as analyzing the entire document. This has significant implications for reducing network traffic and processing time.
  • Naive Bayes Shines: The Naive Bayesian classifier emerged as a strong performer, offering a good balance of accuracy and computational efficiency. Its learning time is linear with the number of examples, and prediction time is constant, making it suitable for real-time applications.
  • ID3's Limitations: Algorithms like ID3, which rely on a small number of features for classification, struggled with this task, suggesting that web page classification requires a more nuanced understanding of multiple features.

Key Results

  • Accuracy: The Naive Bayesian classifier achieved the highest average accuracy across six domains, with an average of 77.1% (standard deviation 4.4%) using 20 training examples. Other algorithms like nearest neighbor and backpropagation also performed well.
  • Efficiency: Analyzing only the first 1024-2048 characters of a page was sufficient to achieve nearly the same accuracy as analyzing the entire document, with a slight decrease from 76.3% to 74.2% on average. Analyzing fewer than 512 characters led to a significant drop in accuracy.
  • Practical Utility: The system demonstrated its ability to suggest interesting links and filter search results, providing a tangible benefit to users in navigating the web.

Limitations and Future Work

Limitations:

  • Initial Data Collection: The system relies on users actively rating pages, which can be a barrier to adoption. The initial setup requires patience and disk space for prefetching.
  • HTML Analysis: The current method of analyzing HTML source can be sensitive to page structure and might misinterpret certain elements (e.g., pointers at the bottom of a page leading to interesting sites).
  • Limited Scope of Features: The initial feature set primarily relied on word presence, potentially missing richer semantic relationships.

Future Work:

  • Improved User Interface: Making the interface more interactive and potentially implementing it as a Netscape plugin to store profiles locally.
  • Ordinal Features: Exploring features that indicate the number of times a word appears, not just its presence or absence.
  • Linguistic and Hierarchical Knowledge: Incorporating stemming, semantic relationships between words (e.g., "protein" vs. "proteins"), and knowledge about word categories (e.g., "gif," "jpg," "jpeg" as image formats) to create more informative features.
  • Summary-Based Rating: Utilizing summaries of web pages (as provided by some search engines) to reduce the need to fetch and parse entire documents.

Summary updated

BibTeX
@inproceedings{1996syskill,
  title = {{Syskill and Webert: Identifying Interesting Web Sites}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}
Figure