Named Entity Recognition in Travel-Related Search Queries

Brooke Cowan, Sven Zethelius, Brittany Luk, Teodora Baras, Prachi Ukarde, Daodao ZhangAAAI 2015 · DOI

TL;DR

This paper tackles the challenge of understanding travel-related search queries by developing a robust Named Entity Recognition (NER) system. They demonstrate that a standard Conditional Random Field (CRF) model, when trained on domain-specific data and augmented with relevant features, can achieve high accuracy in extracting entities like locations and amenities, outperforming previous methods.

What Problem Does This Paper Address?

Search engines are increasingly trying to understand user intent beyond simple keyword matching. Named Entity Recognition (NER) is a crucial step in this process, aiming to identify and classify specific entities within a query (e.g., "hotel in [LOCATION mountain view] with [AMENITY pool]"). However, applying NER to search queries is particularly difficult due to their brevity, lack of context, and reliance on informal language. Traditional NER models, often trained on well-edited text like news articles, struggle with these characteristics, especially features like capitalization which are unreliable in queries. This paper focuses on the travel domain, where understanding entities like locations, hotels, and amenities is vital for providing relevant search results.

Methods

The authors employ a linear-chain Conditional Random Field (CRF) model, a well-established probabilistic sequence model for NER. What's novel here is their application and adaptation of this model specifically to the unique challenges of travel search queries. They train and evaluate the CRF directly on query data, rather than relying on external search engine calls for context, making their system more efficient at test time.

Key to their success is the careful selection and engineering of domain-specific features. These include:

  • Word-based features: Current word, surrounding words, and their combinations.
  • Part-of-Speech (POS) tags: Derived from a domain-specific POS tagger.
  • Word clusters (WC): Generated using Brown clustering on a large corpus of unlabeled queries, capturing semantic similarity.
  • Gazetteer features (GAZ): Leveraging proprietary lists of domain-specific entities (e.g., hotel names, locations, airlines).
  • Heuristic features (HEUR): Incorporating rules derived from a previous, rule-based system as soft constraints.
  • Aliases, Abbreviations, and Spelling (AAS): Preprocessing steps to handle variations in entity names.

Datasets

The paper introduces newly created datasets for travel-domain NER, consisting of 3,500 manually annotated queries. These queries were sourced from two origins:

  1. External search engines: Queries from users who subsequently completed a transaction on Expedia.
  2. Internal search engine: Queries from company employees.

To address the low representation of amenity entities, they employed a semi-supervised approach using a method from Riloff and Jones (1999) to discover additional queries containing amenities. The final datasets comprised 2,084 training queries, 790 development queries, and 589 test queries.

Key Insights and Ideas

  • Domain-specific data is king: Standard NLP models trained on general text perform poorly on queries due to a distributional mismatch. Training and testing on in-domain query data is crucial for high performance.
  • CRFs are surprisingly effective for queries: Despite arguments that sequence models are ill-suited for queries, this paper shows a standard CRF can achieve excellent results with the right features.
  • Feature engineering is paramount: The combination of word clusters and gazetteer features, along with domain-specific POS tags, significantly boosts performance.
  • Heuristics can be soft constraints: Integrating rules from a previous system as features into the CRF model allows for more flexible and effective incorporation of domain knowledge.
  • Capitalization is unreliable: The authors explicitly omit capitalization-based features, acknowledging their low utility in noisy query data.

Key Results

The paper reports significant improvements over a baseline CRF model. Their best performing model achieved an overall entity-level F1 score of 86.4% on the test set, outperforming a baseline score of 82.0%. This improvement was statistically significant. They also observed that the inclusion of a domain-specific POS tagger did not significantly improve F1 scores, suggesting that other features were more impactful. The performance for NAME entities was generally lower than for LOCATION and AMENITY, attributed to longer names and aliasing issues.

Limitations and Future Work

  • NAME entity performance: The authors acknowledge that recovering NAME entities is more challenging, likely due to longer entity lengths and aliasing issues that hinder exact gazetteer matches. Future work will focus on phrase-based cluster features and improved gazette matching for names.
  • Data size: The training set is relatively small. The learning curve suggests that adding more data would yield further gains, but at a diminishing rate.
  • Future data acquisition: They plan to explore crowdsourcing for generating labeled training sets to scale up data acquisition more efficiently.
  • Broader entity coverage: While this paper focuses on LOCATION, NAME, and AMENITY, their system supports 14 travel concepts, and future work could extend the probabilistic approach to more of these.

Summary updated

BibTeX
@inproceedings{cowan2015named,
  author = {Brooke Cowan and Sven Zethelius and Brittany Luk and Teodora Baras and Prachi Ukarde and Daodao Zhang},
  title = {{Named Entity Recognition in Travel-Related Search Queries}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i2.19050},
}
Figure