A Platform to Evaluate the Technology for Service Discovery in the Semantic Web

Cécile Aberg, Johan Aberg, Patrick Lambrix, Nahid ShahmehriAAAI 2006

Introduction

The Semantic Web vision describes a future where software agents autonomously discover, combine, and execute web services using machine-understandable descriptions. However, a significant "chicken and egg" problem persists: while researchers develop sophisticated discovery algorithms and reasoners, the lack of large-scale, semantically annotated web resources prevents rigorous real-world testing. Without a standardized environment to stress-test these technologies, it is difficult to predict their scalability or effectiveness in a production-grade Web. This paper introduces a modular simulation and evaluation platform designed to bridge this gap, providing a "proving ground" for Semantic Web architectures.

Platform Architecture

The authors propose a multi-layered platform built on the Jade (Java Agent DEvelopment Framework). The architecture is designed to be "pluggable," allowing researchers to swap matchmaking algorithms, reasoners, or ontologies to compare performance under controlled conditions.

Overview of the simulation and evaluation platform architecture, spanning from use case assumptions to the final simulation environment.

The platform organizes the Semantic Web ecosystem into four core components:

  1. Web Resources: The underlying content or services being modeled.
  2. Machine-Understandable Data: The semantic layer, including annotations, queries, and ontologies.
  3. Language Specifications: The formal grammars used for description (e.g., OWL, RDF).
  4. Operations: The dynamic interactions between software agents.

Simulation and Multi-Agent System

At the heart of the platform is a Multi-Agent System (MAS) that models the behavior of users and service providers. By assigning specific roles to agents, the system can simulate complex discovery workflows and identify operational bottlenecks.

Detail of the simulation environment, illustrating the interactions between Requesters, Service Managers, and Monitoring Agents.

Key agent roles include:

  • Requesters: Agents that formulate and dispatch semantic queries.
  • Service Managers: Directory agents that store and manage service descriptions.
  • Ontology Agents (OA): Specialized agents that perform logical reasoning (e.g., subsumption checks to determine if a specific service satisfies a general query).
  • Monitor Agents: Tools that observe system events to calculate performance metrics such as response time and precision.

Experimental Setup: The Travel Scenario

To validate the platform, the authors developed a benchmark based on a real-world travel scenario. The dataset included 183 semantically described services from authorized travel providers. The evaluation utilized 14 real-world queries collected from employees, categorized into "simple" (single-service) and "complex" (requiring composition) tasks.

Key Insights: The Scalability Wall

Initial testing revealed a critical bottleneck in pure logic-based discovery. Using a standard Description Logic (DL) approach with the Racer reasoner, the system took approximately 14 minutes to process queries for just 183 services. Monitoring tools identified the cause: each query triggered roughly 300 subsumption operations. This "scalability wall" demonstrated that pure logical reasoning at query time is currently impractical for real-time web applications.

To address this, the authors pivoted to a hybrid approach. Instead of performing heavy reasoning during the search, they designed an algorithm that decomposes semantic descriptions and indexes them in a SQL database at the time of service advertisement. This shifted the computational load to the pre-processing phase, significantly reducing wait times for the end-user.

Results and Performance Analysis

The transition to a hybrid SQL-indexing approach yielded dramatic improvements:

  • Response Time: Average query processing time dropped from 14 minutes to just 10 seconds.
  • Accuracy: The system achieved 100% precision and recall for simple queries.
  • Composition Gap: For complex queries requiring service composition, the system had a 0% success rate. This was an expected result that highlighted the lack of a composition engine in the current matchmaker, providing a clear target for future research.
  • Hardware Constraints: The study also noted that reasoners like Racer are highly resource-intensive, often causing system freezes on hardware with limited CPU or RAM.

Conclusion

The proposed platform provides a vital tool for the Semantic Web community, enabling the identification of performance bottlenecks before large-scale deployment. By demonstrating the necessity of hybrid indexing over pure logical reasoning, the authors provide a practical path forward for scalable service discovery. Future work will focus on integrating service composition engines and expanding the platform's library of default agent behaviors to support broader research adoption.

Summary updated

BibTeX
@inproceedings{aberg2006platform,
  author = {Cécile Aberg and Johan Aberg and Patrick Lambrix and Nahid Shahmehri},
  title = {{A Platform to Evaluate the Technology for Service Discovery in the Semantic Web}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure