TL;DR
This paper introduces the concept of "softbots" – intelligent agents that leverage existing web tools to act on behalf of users. The authors advocate for a "useful first" approach, prioritizing the deployment of functional agents on the web to drive AI research, rather than focusing solely on theoretical intelligence. This strategy has led to the development of practical web agents like MetaCrawler, Ahoy!, and ShopBot, demonstrating the power of AI in real-world applications.
What Problem Does This Paper Address?
The authors frame the World Wide Web as an "information food chain," with raw web pages at the bottom and increasingly sophisticated information processing services higher up. They observe that current popular web services, like Alta Vista, are "information herbivores" that index vast amounts of data but offer homogenized, low-intelligence service due to computational constraints. The paper addresses the challenge of creating truly intelligent, personalized "agents" that can act on behalf of users in this complex digital environment. The authors argue that the web presents a unique opportunity to revitalize AI by tackling practical problems and demonstrating its utility, moving beyond theoretical constructs. They highlight the web community's expectations for robustness, speed, and tangible added value, which pose significant constraints on AI development.
Methods
The core methodology presented is the development and deployment of "softbots" – software robots that act as intelligent agents. These softbots are characterized by their ability to use existing software tools and services on a user's behalf. The paper emphasizes a "useful first" approach, where the focus is on building functional, deployed agents that provide immediate value, rather than pursuing abstract notions of intelligence. This iterative approach allows for the gradual incorporation of AI techniques as the agents evolve.
Key techniques and softbots discussed include:
- MetaCrawler: A unified interface for searching multiple web herbivores (search engines) in parallel. It handles differences in herbivore capabilities by either directly invoking phrase searching or downloading and performing local phrase searches. This demonstrates a meta-interface approach that decouples services from their interfaces.
- Ahoy! The Home Page Finder: This softbot specializes in locating personal home pages by filtering MetaCrawler output. It uses knowledge of web geography and home page appearance, and crucially, employs a heuristic filtering algorithm that learns from user feedback. This "bootstrapping" architecture uses user interaction to generate labeled data for machine learning.
- ShopBot: A softbot designed for comparison shopping on the web. Unlike many previous agents, ShopBot autonomously learns to extract product information from vendor websites by analyzing their URLs and product presentation formats. It leverages a learning algorithm inspired by the Internet Learning Agent (ILA), querying stores for product information and analyzing responses to extract attributes like price.
Datasets
Not discussed in the paper. The paper focuses on the process of acquiring and using data (e.g., user feedback for Ahoy!, product information for ShopBot) rather than pre-defined datasets.
Key Insights and Ideas
- The "Information Food Chain" Metaphor: This provides a clear conceptual framework for understanding the hierarchy of web information services and the potential for agents to operate at higher levels.
- "Useful First" Approach: This pragmatic strategy is presented as a way to overcome the gap between AI prototypes and deployed systems. By focusing on immediate utility and robustness, researchers can drive AI innovation through real-world constraints and user demands.
- Softbots as Intelligent Agents: The paper defines softbots as agents that use tools, highlighting tool use as a hallmark of intelligence. This broadens the scope of AI beyond traditional symbolic reasoning.
- Decoupling Services and Interfaces: MetaCrawler exemplifies how a meta-interface can abstract away the complexities of individual web services, making them more accessible and customizable.
- Harnessing User Feedback for Learning: Ahoy! demonstrates a powerful method for acquiring labeled data in unlabeled environments by leveraging user interaction and feedback. This is crucial for applying machine learning to web-based tasks.
- AI as 99% Computer Science, 1% AI: This provocative statement emphasizes the importance of engineering, robustness, and integration within the broader context of computer science for successful AI deployment.
Key Results
- MetaCrawler: Provides a unified interface to nine popular web search engines, eliminating the need for users to manage multiple search interfaces. It is described as being lightweight enough to run on an average PC and serve as a personal assistant. It currently receives close to 100,000 hits a day.
- Ahoy!: Specializes in finding home pages, typically reducing the number of references by a factor of forty while maintaining high accuracy. It learns to locate home pages even before they are indexed by other search engines. Ahoy! began receiving hundreds of queries per day immediately after its release.
- ShopBot: Enables comparison shopping for software. In a preliminary user study, ShopBot users were able to shop four times faster and find better prices than users relying solely on a web browser.
Limitations and Future Work
- Intelligence Level: The authors acknowledge the potential pitfall of failing to incorporate "substantial intelligence" into their softbots, despite the "useful first" approach.
- Cost vs. Benefit: They raise the question of whether the cost of deploying softbots on the web outweighs the benefits.
- Future Work: The paper suggests that the "useful first" approach will continue to drive new research challenges and innovations. Specific areas for future work include investigating on-line, real-time document clustering for MetaCrawler and developing more sophisticated mixed-initiative dialog for user searches. The authors are committed to making their softbots more intelligent while maintaining usability and usefulness.
Summary updated