Abstract
There is a growing interest in using intelligent search agents (ISAs) in e-commerce and online businesses worldwide in recent years. This interest results from the availability of various sophisticated and powerful intelligent agents that can automate the process of searching through and evaluating reams of information on the Web. While efforts have been made recently to develop various powerful ISAs and multi-agent systems, very little is understood and known about evaluation of such agents. In this paper we describe a simple experimental setup ('system') that can be used to evaluate ISAs without using any complex algorithm and mathematical analysis. The idea is to evaluate search agents based on a performance metric in a controlled environment using complex search queries. For an efficient evaluation of ISAs, we introduce a new metric called 'search speed' which is a ratio of the number of results returned per second per query to the round trip time. This paper provides an in-depth performance comparison of four selected ISAs (Copernic, FirstStop Websearch, iMeta, and WebFerret). These search agents were selected based on their availability, popularity, and interesting characteristics and features. The analysis and empirical results reported in this paper provide some insights into the evaluation of ISAs which may help researchers to evaluate similar search agents.
Introduction
As the volume of information on the World Wide Web continues to expand, the role of Intelligent Search Agents (ISAs) has become increasingly vital. These software tools automate the labor-intensive tasks of searching, filtering, and evaluating data across disparate sources. Despite the proliferation of powerful ISAs, the academic and professional communities lack standardized, accessible frameworks for their evaluation. Existing methodologies often lean toward extreme simplicity or prohibitive mathematical complexity.
This study addresses this gap by proposing a straightforward experimental framework designed to quantitatively benchmark ISAs. By introducing a novel metric, "Search Speed," the authors provide a practical means to determine which agents perform optimally under controlled conditions using complex, real-world queries.
Methodology
The proposed evaluation system shifts the focus from theoretical algorithmic analysis to empirical performance measurement. The framework integrates search agents, performance metrics, and experimentation into a cohesive evaluation cycle.
Central to this methodology is the Search Speed ($S_{speed}$) metric, defined as:
$$S_{speed} = \frac{\text{Total number of search results returned}}{\text{Round Trip Time (RTT) in seconds}}$$
To implement this system, the researchers utilized a standard hardware configuration (Pentium 4 PC with ADSL) and leveraged two primary monitoring tools:
- Wireshark: Employed for packet analysis to capture network throughput and precise Round Trip Times (RTT).
- Windows Task Manager: Used for real-time monitoring of memory consumption (KB) during agent operation.
Experiments were conducted at three distinct times (9 am, 4 pm, and 9 pm) to account for network variability, with each query repeated ten times per agent to ensure statistical robustness.
Experimental Setup and Datasets
The study utilized the live Web as its testing environment rather than a static dataset. Four search agents were selected for evaluation based on their popularity and feature sets: Copernic Agent v6.2, FirstStop Websearch 5, iMeta Search 5, and WebFerret 6.0.
The agents were tested against four specific queries of increasing complexity:
- 4 terms: "How to make pudding"
- 5 terms: "Who said: 'now or never'"
- 6 terms: "Information on colleges located in Germany"
- 7 terms: "Information on effects of caffeine on heart"
Results and Discussion
The empirical analysis revealed several critical insights into ISA performance. A primary observation was the "Complexity Penalty": as query complexity increased, particularly with the use of exact phrases, search speeds tended to decrease significantly. This suggests that the overhead of specific filtering remains a challenge even for "intelligent" agents.
Furthermore, a clear trade-off between performance and resource consumption was identified. The most performant agent in terms of speed often required the highest system resources.
Key performance highlights include:
- Speed and RTT: iMeta was the top performer, achieving a search speed of 8.4 results/sec/query and the lowest RTT of 5.1 seconds.
- Resource Efficiency: WebFerret was the most lightweight, consuming less than 4000 KB of memory, whereas iMeta exceeded 9000 KB.
- Result Volume: Copernic returned the highest average number of results (65.1), though it did not match iMeta's speed.
A one-way ANOVA (F-test) yielded an F-value of 20.99, confirming that the differences in search speeds between the agents were statistically significant and that the $S_{speed}$ metric is a valid differentiator.
Conclusion
This research provides a practical roadmap for evaluating Intelligent Search Agents without the need for complex mathematical modeling. While the study was limited to four free agents and focused on quantitative rather than qualitative (accuracy/usability) metrics, it establishes a foundation for future benchmarking. Future work should expand this framework to include commercial ISAs and explore performance across different media types, such as image and video search.
Summary updated