Query-aware Tip Generation for Vertical Search

Yang Yang, Junmei Hao, Canjia Li, Zili Wang, Jingang Wang, Fuzheng Zhang, Rao Fu, Peixu Hou, Gong Zhang, Zhongyuan Wang

Abstract

Vertical search engines often provide short "tips" to help users make quick decisions. However, traditional query-agnostic tips often fail to align with specific user intents, creating a semantic gap between search results and user needs. This paper introduces a query-aware tip generation framework that integrates query information into both the encoding and decoding phases of neural models. Evaluations on a large-scale real-world dataset from the Dianping app demonstrate significant improvements in tip relevance and a measurable boost in click-through rates (CTR) in a production environment.

Introduction

In vertical search platforms like Yelp, TripAdvisor, or Dianping, "tips"—concise highlights extracted from user reviews—serve as critical decision-making aids. They allow users to grasp the essence of a business without wading through lengthy reviews. However, a significant limitation of current systems is their query-agnostic nature. A user searching for "Steak" might be presented with a tip about a restaurant's "excellent dessert," which, while positive, fails to address their immediate intent. This "semantic gap" reduces the utility of search results.

Examples of query-aware tips for "Steak" and "Ramen" searches on the Dianping app, illustrating how tips are tailored to the user's specific query.

To bridge this gap, researchers have developed a framework for query-aware tip generation, ensuring that the displayed highlights are dynamically tailored to the user's specific search terms. By making tips responsive to the query, the system helps users find the information most relevant to their current search context.

Methodology

The authors propose a query-aware encoder-decoder framework and provide two specific implementations: one based on the Transformer architecture and another based on Recurrent Neural Networks (RNNs).

Transformer-based Adaptation

The Transformer model was modified to incorporate query information at multiple stages. A "Review-aware Query Encoder" uses multi-head attention to let the query interact with the review text early in the process. This provides vital context to often-short search queries (e.g., "cake"). Subsequently, a "Query-aware Review Encoder" processes the review using this contextualized query information. Finally, the "Query-aware Tip Decoder" directly accesses both the query representation and the review hidden states during the generation phase to ensure the output remains relevant to the user's intent.

The proposed Transformer-based architecture, showing how query information is integrated into the encoding and decoding layers.

RNN-based Adaptation

For the RNN (Bi-LSTM) implementation, the authors introduced a "selective gate" network. This mechanism acts as a soft filter, identifying and emphasizing tokens in the review that align with the query while suppressing irrelevant content. This mimics the human behavior of skimming a long review specifically looking for mentions of a particular topic, such as "parking" or "atmosphere."

Experimental Evaluation

The models were evaluated on two distinct datasets:

  1. Debate: A public English dataset from Debatepedia containing ~13,700 triplets of query, document, and summary, serving as a benchmark for query-based summarization.
  2. Dianping: A massive, newly created in-house dataset comprising 224,730 tuples (POI, Review, Query, Tip). This dataset was constructed using BM25 weighting to align historical search logs with relevant reviews to create a ground-truth dataset for training.

Key Results

The Transformer-based model ("Trans + Both") consistently outperformed other variants in both automatic and manual evaluations:

  • Automatic Metrics: On the Dianping dataset, the query-aware Transformer reached a BLEU score of 54.20, significantly higher than the query-agnostic baseline of 48.71.
  • Manual Evaluation: Human annotators rated the query-aware tips higher in Relevance (63.72% vs 39.25% for the baseline) and overall Usefulness.
  • Online Impact: A real-world A/B test on the Dianping app, serving approximately 10 million daily queries, showed a measurable increase in Click-Through Rate (CTR). While the absolute increase was 0.06%, at this scale, it represents a substantial boost in user engagement.

Conclusion and Future Directions

The transition from query-agnostic to query-aware tip generation represents a significant step forward in making vertical search more responsive to user intent. The Transformer-based approach proved particularly effective due to its ability to handle long-range dependencies in user reviews.

Future work aims to incorporate user personalization, moving beyond query-awareness to generate tips tailored to individual user profiles (e.g., preferences for spicy food or budget-friendly options). Additionally, the authors plan to explore ways to maintain high-quality generation even for businesses with limited review data.

Summary updated

BibTeX
@inproceedings{yang2020query,
  author = {Yang Yang and Junmei Hao and Canjia Li and Zili Wang and Jingang Wang and Fuzheng Zhang and Rao Fu and Peixu Hou and Gong Zhang and Zhongyuan Wang},
  title = {{Query-aware Tip Generation for Vertical Search}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2020},
  doi = {10.1145/3340531.3412740},
}
Figure