Abstract
This work falls in the areas of information retrieval and semantic web, and aims to improve the evaluation of web search tools. Indeed, the huge number of information on the web as well as the growth of new inexperienced users creates new challenges for information retrieval; certainly the current search engines (such as Google, Bing and Yahoo) offer an efficient way to browse the web content. However, this type of tool does not take into account the semantic driven by the query terms and document words. This paper proposes a new semantic based approach for the evaluation of information retrieval systems; the goal is to increase the selectivity of search tools and to improve how these tools are evaluated. The test of the proposed approach for the evaluation of search engines has proved its applicability to real search tools. The results showed that semantic evaluation is a promising way to improve the performance and behavior of search engines as well as the relevance of the results that they return.
Introduction
Information Retrieval (IR) systems are incredibly efficient at finding documents that contain specific keywords. However, they often struggle with semantics—the actual meaning and intent behind the user's query and the vocabulary used in the documents. For example, if a user searches for "automobile," a purely keyword-based system might rank a highly relevant document lower simply because it uses the word "car" instead. This paper addresses the challenge of improving the selectivity and evaluation of existing search tools by injecting a layer of semantic understanding into the ranking process, ensuring that documents conceptually related to the query are prioritized even if exact keyword matches are absent.
Methodology
The proposed architecture acts as a semantic wrapper around existing search engines. The pipeline consists of several distinct modules:
- Search & Extraction (SM & IEM): The system passes the user's query to Google, Yahoo, and Bing, retrieving the top 20 results. It parses these pages to extract titles, abstracts, and URLs, stripping away irrelevant HTML tags.
- Semantic Projection (SPM): Instead of raw query words, the system projects the query onto the WordNet 3.0 ontology. It extracts synonyms and hypernyms to build an expanded "semantic vector."
- Calculation (CM): Using the Vector Space Model (VSM), the system calculates term weights using a modified TF-IDF formula and measures the Cosine Similarity between the expanded semantic query vector and the document vectors.
- Re-ranking (RM & PM): The system re-orders the original results based on these new semantic similarity scores, presenting a conceptually refined list to the user.
Evaluation and Results
The system was evaluated using live web data and the WordNet 3.0 linguistic resource. The authors generated a custom evaluation set by running 25 specific search scenarios (15 "simple" everyday queries and 10 "complex" specialized queries), resulting in a pool of 500 web pages evaluated via a manual scoring grid.
Key findings include:
- Overall Improvement: Semantic ranking improved the overall average effectiveness score for both Google (from 7.62 to 8.29 out of 10) and Yahoo (from 6.93 to 7.02).
- Complex vs. Simple Queries: The performance gap between classical and semantic ranking widened for complex queries. For Google, semantic ranking added 0.67 points for simple queries, but 0.75 points for complex ones, demonstrating that semantic understanding provides the most significant boost where traditional engines struggle.
- Relevance Gains: When looking strictly at results relevance, Yahoo saw a jump from 5.06 (classical) to 7.66 (semantic).
Discussion and Conclusion
The study demonstrates that ontologies can be effectively "bolted onto" existing search systems. By using a meta-search approach that re-ranks top results, the system leverages the massive indexing power of major engines while adding nuanced linguistic understanding.
However, there are trade-offs. Semantic expansion slightly increased the rate of redundant results because the broadened vocabulary matches more pages from the same domains. Additionally, processing overhead currently limits the system to evaluating the top 20 results. Future work could involve swapping general-purpose ontologies like WordNet for domain-specific ones (e.g., medical or legal) to create highly specialized semantic search wrappers.
Summary updated