Research on the Database Process Using GRID Technologies

Yang YangAdvances 2015 · DOI

Abstract

Grid technology is widely used in recent years, it has solved the real sharing of resources, make each node of unified command and use of resources, the information grid is on the basis of computational grid, using data mining. Information fusion and search engine technology and building are advantageous for the collection and sharing of grid resources. The goal to create a build on OS and Web is based on the new generation of Internet information platform. In this platform, the distribution of the information processing is the collaboration and intelligent. Focus crawler sets according to the target subject. In the form of intelligent collection topics come from the Web page, and then adopt the method of machine learning. The information retrieval on the collected information intelligent need process and analyze. Finally, the effective retrieval way is to meet the demand of the user's information retrieval.

Introduction

Despite the vastness of the modern internet, data often remains trapped in "information islands"—siloed resources that are difficult for users to navigate or aggregate. Traditional search engine crawlers typically use breadth-first traversal, which aims for quantity over quality; they try to index as much of the web as possible but often struggle with precision for niche topics.

This research explores the intersection of Grid computing and intelligent information retrieval. By leveraging the Open Grid Services Architecture (OGSA) and a specialized "Super-ZT-Spider" topic crawler, the proposed distributed system aims to break down these information islands and provide highly relevant, subject-specific search results.

Grid Architecture: The Sandglass Model

The framework is built upon the "five-layer sandglass structure" common in Globus-based grid systems. Much like the TCP/IP stack, this model centers on a core "agreement" layer. It moves from the physical "Fabric" (Structure) layer up through "Connectivity" (Connect), "Resource," and "Collective" (Convergence) layers to the "Application" layer.

The five-layer sandglass structure of Grid architecture compared to standard network layers.

This architecture allows heterogeneous resources to interact through standardized protocols like SOAP (Simple Object Access Protocol) and WSDL (Web Services Description Language). Grid technology acts as an extension of existing networks, adding a layer of "intelligent collaboration" rather than replacing current architectures.

The Super-ZT-Spider: Purposeful Crawling

The core innovation is the "Super-ZT-Spider," a focus crawler designed for purposeful information retrieval. Unlike general crawlers, it uses a specific similarity formula to calculate the correlation between a crawled page and a target subject. By using machine learning to evaluate the "anchor text" of hyperlinks before clicking them, the crawler avoids the inefficiency of downloading irrelevant pages.

Queue Management and State Transitions

To handle the heavy load of web crawling, the system implements a database-backed four-queue system: Waiting, Processing, Error, and Complete. This ensures that the crawling process is robust and can handle errors gracefully.

The state transition diagram for URL management within the Super-ZT-Spider crawler, illustrating the flow from discovery to completion.

The crawler integrates results from multiple "Main Spiders" and uses a feedback loop to adjust parameters dynamically. By distributing the crawling process across multiple Grid nodes, the system prevents a single crawler from getting "stuck" in a specific area of the web (a local optimum), ensuring a more comprehensive search of the topic space.

Key Insights and Results

The integration with Globus Toolkits demonstrates that grid services can effectively manage tasks like security, resource location, and communication for complex data processing jobs. Key qualitative results include:

  • Improved Precision: The Super-ZT-Spider algorithm strengthens the classification effect of web pages by studying the correlation between parent and child nodes.
  • Dynamic Feedback: Adjacent nodes share correlation information, allowing the crawler to "learn" and adapt its strategy online.
  • Standardization: The use of OGSA is essential for making these distributed components interoperable.

Limitations and Future Work

While promising, the system faces several challenges:

  • Training Overhead: Neural network-based crawlers require significant offline training and a large volume of manual samples, which can be a bottleneck.
  • Lack of Online Learning: Many existing models still lack robust "online learning" capabilities to adapt to the web in real-time without prior training.
  • Quantitative Validation: Future work will involve large-scale deployment to measure precision/recall curves and speedup benchmarks, providing the rigorous experimental data currently missing.

Summary updated

Abstract

Grid technology is widely used in recent years, it has solved the real sharing of resources, make each node of unified command and use of resources, the information grid is on the basis of computational grid, using data mining. Information fusion and search engine technology and building are advantageous for the collection and sharing of grid resources. The goal to create a build on OS and Web is based on the new generation of Internet information platform. In this platform, the distribution of the information processing is the collaboration and intelligent. Focus crawler sets according to the target subject. In the form of intelligent collection topics come from the Web page, and then adopt the method of machine learning. The information retrieval on the collected information intelligent need process and analyze. Finally, the effective retrieval way is to meet the demand of the user's information retrieval.
BibTeX
@inproceedings{yang2015research,
  author = {Yang Yang},
  title = {{Research on the Database Process Using GRID Technologies}},
  booktitle = {Advances},
  year = {2015},
  doi = {10.2991/iccse-15.2015.58},
}
Figure