TL;DR
This paper tackles the challenge of automatically identifying the genre of web pages (like blogs, wikis, and forums) without needing to understand the language of the content. They achieve this by focusing on the structural patterns within a page's HTML, demonstrating that this approach is highly effective and language-agnostic.
What Problem Does This Paper Address?
The web is exploding with user-generated content, making it crucial for search engines and other tools to understand the type of content they're dealing with, not just its topic. This "genre" (e.g., is it a blog post, a forum discussion, or a wiki page?) dictates how users expect to interact with it and how it's structured. Existing methods often rely on linguistic analysis, which is problematic as the web becomes increasingly multilingual. This paper aims to bridge this gap by developing a language-independent method for web genre detection, specifically focusing on popular social web genres.
Methods
The authors propose a novel approach that leverages the hierarchical structure of HTML mark-up, moving away from content-based linguistic analysis. Their core innovation lies in identifying "structural patterns" – recurring mark-up structures within a page that are indicative of its genre.
Here's a breakdown:
- Feature Engineering: They combine traditional, language-independent features (like HTML tag frequencies, URL properties, and text-to-markup ratios) with their novel "structural pattern" features.
- Structural Pattern Extraction: This method analyzes the HTML Document Object Model (DOM) to find repeating sub-tree structures. These patterns are identified by comparing the structural representations of nodes and their sub-trees. Features are then computed based on the number, size, hierarchy, location, and content ratio of these identified patterns.
- Classification: They use Support Vector Machines (SVM) with Sequential Minimal Optimization (SMO) for classification.
Datasets
The authors compiled their own corpus for this research due to the lack of suitable multi-lingual, genre-specific datasets.
- Corpus: A custom corpus was created containing 1000 multi-lingual instances across five genres/sub-genres (Blog_Page, Blog_Post, Wiki_Page, Forum_Start, Forum_Thread).
- Language Distribution: The corpus includes approximately 65% English, 7% German, 7% French, and content in about twenty other European and Asian languages.
- Diversity: Care was taken to include pages from various popular web applications within each genre (e.g., different wiki engines like MediaWiki and MoinMoin).
Key Insights and Ideas
- Structure Over Semantics: The most significant insight is that the structural layout of HTML is a powerful, language-independent signal for genre detection. This bypasses the complexities of natural language processing in a multilingual context.
- Recurring Patterns are Key: The "structural patterns" – repeated mark-up blocks – are highly informative. The paper highlights that the ratio of content contained within these recurring patterns is a strong predictor of genre.
- Sub-genre Nuances: Differentiating between closely related sub-genres (like a blog's main page vs. a specific blog post) is the most challenging aspect, suggesting that finer-grained structural distinctions are needed for precise classification.
- Efficiency: Their approach uses a relatively small set of features (144) compared to many linguistic methods, leading to reduced computational complexity.
Key Results
- High Accuracy: Using their combined feature set, the authors achieved an overall accuracy of 90.8% for classifying the defined web genres.
- Sub-genre Challenge: The primary source of misclassification was distinguishing between blog start pages and blog post pages. When these were grouped into a broader "blog" category, the accuracy increased to 95.1%.
- Important Features: Feature importance analysis revealed that HTML tag frequencies, URL analysis, link analysis, HTML facets, and the proposed pattern ratio features were among the most discriminative.
Limitations and Future Work
- Outlier Detection: A key limitation acknowledged is the current focus on a predefined set of genres. Future work will involve developing methods to detect "outliers" – pages that don't fit into any of the known genres. Preliminary results show that even with an outlier class, accuracy remains high (86%).
- Broader Genre Set: The authors plan to extend their approach to a wider range of web genres.
- Community Mining Application: The results are intended to be applied to the field of Community Mining.
Summary updated