TL;DR
The "Yakamoz" system is a specialized distributed crawler designed specifically for real-time opinion mining. Unlike general search engine crawlers that prioritize global ranking, Yakamoz focuses on high-interaction sites (forums, microblogs) and uses a master-slave architecture to handle dynamic content like AJAX-driven news comments.
What Problem Does This Paper Address?
Standard web crawlers are built for search engines, where the goal is to index the entire web and rank pages based on general authority (like PageRank). However, opinion mining—tracking public sentiment on specific events—requires a different approach.
The authors identify three main challenges in this niche:
- Targeting: Opinion mining needs to focus on "hot" spots like forums and blogs rather than static corporate pages.
- Dynamic Content: Modern sentiment often lives in comment sections or microblogs that load via AJAX, which simple HTML crawlers cannot see.
- Timeliness: The system must adjust its crawling frequency based on how fast a discussion is evolving, requiring a feedback loop between the analysis engine and the crawler.
Methods
The authors propose Yakamoz, a distributed master-slave system:
- Master Node (Crawler Manager): Acts as the brain. It handles strategy parsing (site-specific rules), URL management using a Bloom filter for deduplication, and a five-level priority queue for task scheduling. It also maintains a backup node to ensure system resilience.
- Slave Nodes (Crawlers): These are the "workers." Each slave contains a Downloader Factory that can spin up three types of downloaders:
- Default: For fast, static HTML.
- Login: For accessing forums or social networks requiring authentication.
- Webkit: A headless browser (based on QT) that renders dynamic pages and AJAX calls, essential for capturing news comments.
- Feedback Loop: The system provides interfaces that allow an external opinion analysis system to send feedback, adjusting crawling strategies in real-time based on the "heat" of a topic.
Datasets
The paper does not use a static, pre-existing dataset. Instead, it describes a system designed to collect data from:
- Major news websites (specifically targeting comment sections).
- Forums and blogs.
- Microblogs (e.g., Weibo or Twitter-like platforms).
Key Insights and Ideas
- Vertical vs. Horizontal Crawling: The authors argue that for opinion mining, "vertical" crawling (depth and specificity) is more valuable than "horizontal" crawling (breadth).
- Resource Management for AJAX: Because rendering pages with Webkit is resource-intensive, the system limits it to one instance per slave, running threads serially to prevent system crashes while still capturing dynamic data.
- Smart Scheduling: Tasks are split into fragments of 100-200 URLs. The master node uses "heartbeat" packets to monitor slave health; if a slave fails to report back within a specific window, its task is automatically reassigned to another node.
- Hashing for Load Balancing: URLs are grouped by hash values in the priority queue. This ensures that the system doesn't accidentally "DDoS" a single site by having all crawlers hit it simultaneously.
Key Results
- Versatility: The system successfully supports diverse site architectures, from simple news sites to complex, interactive microblogs.
- Efficiency: By incorporating parameters like "keyword count" and "published time" into the refreshing strategy, the system monitors active discussions more frequently than stagnant ones, improving the "freshness" of the data.
- Reliability: The master-slave architecture with periodic database syncing allows the system to resume quickly after a node failure without losing the crawl frontier.
Limitations and Future Work
- Webkit Bottleneck: The Webkit downloader is identified as a significant resource hog. The current implementation only allows one instance per system, which could limit throughput on highly dynamic sites.
- Algorithm Detail: While the paper mentions using keyword counts and publication times to adjust strategies, the specific mathematical models or weights for these parameters are not detailed in depth.
- Future Work: The authors suggest further refining the feedback mechanism between the analysis system and the crawler to automate the discovery of "hot" topics without manual configuration.
Summary updated