Abstract
Supervised text classification is often hindered by the high cost and time required for manual data labeling. This paper introduces a fully automatic framework that leverages web search engines to generate training data without human intervention. By utilizing "common" and "discriminative" concept expansion techniques, the framework refines search queries to produce high-quality, distinct training sets. Our findings indicate that search engine snippets provide a superior, low-noise alternative to full web documents, enabling unsupervised classifiers to achieve performance levels (up to 81% accuracy) competitive with supervised models.
Introduction
The primary bottleneck in modern text classification is the reliance on large, manually labeled datasets. This "data bottleneck" is particularly problematic in dynamic fields where concepts evolve rapidly, rendering static datasets obsolete. While the web offers a vast repository of potential training data, simple keyword-based retrieval often suffers from ambiguity (e.g., "Apple" as a fruit vs. a company) and a lack of discrimination between similar classes (e.g., "Java" vs. "Python" in a programming context).
Methodology
To overcome these challenges, we propose a dual-strategy for concept expansion to refine search queries and gather better training data:
- Expansion by Common Concepts (CM): This technique resolves ambiguity by identifying the shared context within a set of target classes. By situating a class like "Architecture" within a set including "Programming" and "Theory," the system can identify "Computers" as a common ancestor (using directories like DMOZ), refining the search to exclude irrelevant domains like civil engineering.
- Expansion by Discriminative Concepts (DM): This strategy focuses on uniqueness. An iterative algorithm identifies terms that are highly similar to a specific "host" class but dissimilar to others. For example, "iPod" serves as a discriminative concept for "Apple" when compared to "Microsoft."
Search Engine Selection
The effectiveness of this framework depends on the quality of the underlying search engine. As shown in Figure 1, Google significantly outperformed Yahoo! and Live Search across various computer science categories, making it the preferred engine for data acquisition in this study.
Data Sources: The Power of Snippets
A key discovery in our research is that search engine snippets—the brief summaries provided on results pages—are more effective training sources than full web documents. Full documents often contain extraneous "noise," such as HTML boilerplate or tangential content, that can degrade classifier performance. Snippets act as a natural summarization tool, capturing the essence of a concept without the "fluff."
As illustrated in Figure 2, snippet-based training consistently yielded higher average performance scores compared to document-based training across all tested categories, including Architecture, IR, and Network.
Experimental Results
The framework was evaluated on two datasets: a collection of CS Papers (2,500 documents across five categories) and the WebKB dataset.
- CS Papers: The combined CM+DM method achieved an average accuracy of 81%, significantly outperforming baseline methods (57%) and previous state-of-the-art unsupervised models like LiveClassifier (71%).
- WebKB: Despite the inherent noise in web page data, the approach reached 64% accuracy, a 9% improvement over the baseline.
- Efficiency and Hybrid Potential: The use of snippets significantly reduces the computational overhead of downloading and parsing full pages. Furthermore, experiments showed that combining web-mined data with just 20% of a manually labeled dataset allows the model to match the performance of a fully supervised system.
Conclusion
This research demonstrates that intelligently mined web data can effectively replace or augment manual labeling for text classification. By leveraging the inherent structure of search results and applying targeted concept expansion, we can build robust, high-performance classifiers automatically. Future work will focus on refining document selection within the sampled corpus and investigating how to choose the highest quality documents from the sampled Web corpus.
Summary updated