Abstract
Short, "noisy" text classification, such as chat dialogues or search queries, poses a significant challenge for traditional Bag-of-Words (BOW) models due to a lack of internal context. This paper presents a framework for injecting "world knowledge" from diverse external sources—Wikipedia, Yahoo Answers, and the Open Directory Project—to enhance classification performance. By generating a vast pool of semantic features and applying aggressive pruning via Mutual Information, the proposed method achieves a 48-65% error reduction over state-of-the-art baselines, rivaling human performance.
Introduction
Standard text classification typically relies on word frequency within a document. While effective for long-form content, this approach fails for short snippets where context is sparse. For instance, a query like "My focus doesn't start" requires external knowledge to identify "Focus" as a vehicle model rather than a common noun. This research explores how to effectively transfer background knowledge from massive, heterogeneous web sources into specific classification tasks to bridge this semantic gap.
Methodology
The authors propose a three-pronged approach to feature generation:
- Predictions as Features (PAF): Classifiers are trained on labeled external data (e.g., Yahoo Answers categories), and their outputs serve as new features for the primary task.
- Explicit Semantic Analysis (ESA): Text is mapped to high-dimensional Wikipedia concepts, linking terms like "engine" to the broader "Car" concept even if the word "car" is absent.
- Structure-Based Feature Generation (SBFG): Unlabeled data is clustered, and similarity to these clusters provides high-level features.
The "Less is More" Paradox
The combination of these methods generates over 100,000 features. To mitigate noise and overfitting, the authors employ Mutual Information (MI) to rank and select only the most informative features.
As shown in the figure above, accuracy peaks when using a limited subset of features (approximately 200-500 per class). This demonstrates that excessive features introduce detrimental noise that can overwhelm the classification signal.
Experimental Setup
The approach was evaluated on two primary datasets:
- Google Answers: 8,850 short, expert-oriented questions across 9 categories.
- Switchboard Corpus: 6,840 transcribed conversational sentences across 10 topics like "Movies" or "Family Life."
External knowledge was drawn from Wikipedia (200k concepts), Yahoo Answers (822 sub-categories), and the Open Directory Project (494 sub-categories).
Key Results
The integration of world knowledge significantly outperformed traditional BOW baselines. The researchers noted that leveraging multiple heterogeneous sources is superior to relying on a single source like Wikipedia alone.
| Dataset | Baseline (BOW) | Proposed Method (All Sources) | Error Reduction |
|---|---|---|---|
| Google Answers | 46.55% | 89.41% | ~65% |
| Switchboard | 68.16% | 96.88% | ~48% |
The Precision-Recall curves demonstrate that the "All" model, which integrates multiple heterogeneous sources, maintains higher precision across most recall levels compared to BOW and BOW+ESA models.
Discussion and Future Work
A critical insight is that heterogeneity matters; Wikipedia provides broad concepts, while Yahoo Answers offers colloquial context that matches the primary tasks better. While the final classifier is efficient due to the small feature set, the initial stage of generating 100,000+ features requires significant offline preprocessing. Future work may address redundancy in feature selection and explore how well this method adapts to highly specialized technical or medical domains.
Summary updated