Vertical selection in the presence of unlabeled verticals

Fernando Diaz, Jaime Arguello, Fernando Diaz 0001, Jean-François PaiementSIGIR 2010 · DOI

Modern search engines face a significant challenge when integrating specialized "vertical" results—such as news, images, or health—into general web search pages. While supervised machine learning is the standard for deciding when to trigger these verticals, it traditionally requires a massive amount of human-labeled data for every new category added. This paper, presented at SIGIR 2010, proposes a solution to this "cold start" problem by using domain adaptation to build selection models for new, unlabeled verticals using data from existing ones.

The Core Challenge: Data Hunger

Vertical selection is the task of predicting which specialized databases are relevant to a user's query. Traditionally, adding a "Real Estate" vertical would require humans to manually label thousands of queries to train a model. The authors address this by distinguishing between "portable" features (universal signals) and "adaptive" features (category-specific signals), allowing a system to accurately predict relevance for a brand-new vertical without a single new human annotation.

Methodology: Portability and Adaptability

The researchers developed a two-stage approach based on Gradient Boosted Decision Trees (GBDT):

  1. Portability (The Foundation): A base model is built to work across any vertical. This involves Vertical Balancing (weighting training instances to prevent popular verticals from dominating) and Feature Weighting (identifying features that are consistently predictive across all known source verticals).
  2. Adaptability (The Tailoring): The portable model is then tuned to the new, unlabeled target vertical using Self-Training and TRADA (Tree-based Domain Adaptation). The model makes "pseudo-label" predictions on the new data and adds regression trees that focus on "non-portable" features—like specific keywords—that define the unique niche of the target vertical.

Key Insights: The Portability Paradox

A critical finding of the study is that features highly effective for a specific vertical (e.g., the word "weather" for a weather vertical) are often poor for portability. The authors discovered that query-vertical features—which measure the match between a query and a vertical's specific content—are the most portable across domains.

Figure 2: Comparison of feature portability (measured by harmonic average performance) between query-vertical features and query-only features.

As shown in the analysis of feature performance, query-vertical features maintain a much higher and more consistent performance (havg) across different verticals compared to query-only features, which tend to be highly vertical-specific.

Experimental Results

The study utilized a massive real-world dataset from Yahoo! Search covering 11 verticals (including Finance, Games, and Health) and over 25,000 unique queries.

Figure 1: Performance comparison (AP and NDCG) of the proposed TRADA+FW models against the SR baseline across 11 verticals.

The results demonstrate that the "Portable + Adaptive" models (TRADA+FW) significantly outperformed standard unsupervised baselines. In many cases, the unsupervised method achieved a high percentage of the performance of a "cheating" model trained with actual labels. For example, in the "Video" vertical, the proposed method reached nearly 70% of the precision of a fully supervised model without using any video-specific labels.

Conclusion and Future Work

This research proves that vertical selection models can be effectively bootstrapped for new categories using existing data. While the optimal parameters for self-training currently vary by vertical, the framework provides a robust path for search engines to scale their specialized offerings. Future work may involve weighting source verticals based on their similarity to the target vertical to further refine the adaptation process.

Summary updated

BibTeX
@inproceedings{diaz2010vertical,
  author = {Fernando Diaz and Jaime Arguello and Fernando Diaz and Jean-François Paiement},
  title = {{Vertical selection in the presence of unlabeled verticals}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2010},
  doi = {10.1145/1835449.1835564},
}
Figure