Incorporating Semantic Knowledge into Latent Matching Model in Search

Shuxin Wang, Xin Jiang, Hang Li, Jun Xu, Bin Wang

TL;DR

This paper introduces a novel latent matching model for search that enhances performance, especially for "tail" (less frequent) queries, by incorporating semantic knowledge like synonyms and categories alongside traditional click-through data. They propose two optimization methods (coordinate descent and gradient descent) and demonstrate significant improvements in search relevance ranking on real-world app search data.

What Problem Does This Paper Address?

Traditional information retrieval (IR) models often struggle with "term mismatch," where queries and documents use different words to express the same concept. While latent space models have improved search by mapping queries and documents to a shared representation space, they heavily rely on click-through data for training. This poses a significant challenge for "tail" queries and documents, which have insufficient click data to train robust models. The paper aims to bridge this gap by integrating external semantic knowledge (synonyms, categories) into the latent matching model to improve its ability to handle these less frequent but important search scenarios.

Methods

The core of the paper is the extension of a latent matching model (specifically, building upon RMLS - Regularized Mapping to Latent Structures) by incorporating semantic knowledge. This is achieved by adding regularization terms to the objective function. These terms encourage the model's latent representations to align with the provided semantic knowledge.

Key methodological contributions include:

  • Semantic Knowledge Integration: The paper proposes incorporating semantic knowledge (synonyms and categories) through regularization terms in the objective function. This allows the model to learn better query-document mappings, especially when click data is sparse.
  • Optimization Algorithms: Two optimization algorithms are developed to solve the learning problem:
  • Coordinate Descent: An iterative approach that updates individual components of the mapping matrices. It's noted for its fast convergence but can be computationally intensive for high-dimensional latent spaces.
  • Gradient Descent: An alternative that updates the entire matrices. It's computationally lighter per step but generally requires more iterations to converge.
  • Semantic Knowledge Acquisition: The paper outlines methods for acquiring semantic knowledge:
  • Synonym Mining: Automatically mining synonym pairs from click-through data by analyzing query clusters in a bipartite graph.
  • Category/Tag Utilization: Using existing category tags assigned to documents (e.g., app tags) and deriving relevant terms for each tag.

Datasets

The experiments were conducted on two large-scale datasets derived from a mobile app search engine. These datasets consist of query-document pairs and their associated click-through logs.

  • Training Data: Two datasets were created: one from one week of click-through data and another from one month of click-through data.
  • Test Data: Two test datasets were sampled, each containing 200 queries, split into 100 "head" (frequent) queries and 100 "tail" (infrequent) queries. Query-app pairs in the test sets were labeled into four matching levels: Excellent, Good, Fair, and Bad.

The paper notes that these datasets are larger than typical IR datasets, which is beneficial for training latent matching models.

Key Insights and Ideas

  • Semantic Knowledge is Crucial for Tail Queries: The most significant insight is that incorporating semantic knowledge, beyond just click data, dramatically improves performance for tail queries. This is because semantic knowledge can infer relationships that are not evident from sparse click logs.
  • Latent Space Alignment: The paper demonstrates that semantic knowledge effectively "clusters" similar words closer together in the learned latent space, leading to more accurate matching.
  • Regularization as a Bridge: Using regularization terms is an elegant way to bridge the gap between the learned latent representations and external semantic knowledge without requiring a complete overhaul of the model architecture.
  • Trade-offs in Optimization: The choice between coordinate descent and gradient descent depends on the latent space dimensionality and desired efficiency, highlighting practical considerations in model training.

Key Results

  • Significant Performance Gains: The proposed LMM model, especially when augmented with semantic knowledge (LMM-X), consistently outperformed traditional IR models like BM25 and baseline latent models like PLS and RMLS.
  • Tail Query Dominance: Improvements were most pronounced for tail queries, with NDCG scores showing substantial increases compared to head queries. For example, LMM-Both (100) achieved an NDCG@10 of 0.729 on tail queries with one-week data, compared to 0.696 for BM25.
  • LMM Outperforms Baselines: The base LMM model (without semantic knowledge) also showed advantages over RMLS and PLS, indicating the effectiveness of its formulation.
  • Synergistic Effect of Knowledge: Combining both synonym and category knowledge (LMM-Both) generally yielded the best results, with statistically significant improvements over using either type of knowledge alone.
  • Dimensionality Impact: Increasing latent dimensionality generally improved performance, with a plateau observed around 500 dimensions, suggesting a good trade-off between performance and efficiency.

Limitations and Future Work

  • Quality of Semantic Knowledge: The paper acknowledges that the quality of the mined semantic knowledge can be a limitation. Noisy synonyms (e.g., "google" and "baidu" being identified as synonyms) can sometimes lead to suboptimal ranking. Future work should focus on improving the quality and robustness of semantic knowledge acquisition.
  • Non-linear Models: The current work focuses on linear latent matching models. The authors suggest exploring semantic knowledge incorporation into non-linear models as a future direction.
  • More Knowledge Types: Incorporating additional types of knowledge, such as knowledge graphs, is also proposed as a future research avenue to further enhance matching capabilities.
  • Handling Tail Queries: The paper explicitly states that semantic matching on long-tail data remains challenging, and future work will focus on developing models and algorithms specifically for handling tail queries more effectively.

Summary updated

Abstract

The relevance between a query and a document in search can be represented as matching degree between the two objects. Latent space models have been proven to be effective for the task, which are often trained with click-through data. One technical challenge with the approach is that it is hard to train a model for tail queries and tail documents for which there are not enough clicks. In this paper, we propose to address the challenge by learning a latent matching model, using not only click-through data but also semantic knowledge. The semantic knowledge can be categories of queries and documents as well as synonyms of words, manually or automatically created. Specifically, we incorporate semantic knowledge into the objective function by including regularization terms. We develop two methods to solve the learning task on the basis of coordinate descent and gradient descent respectively, which can be employed in different settings. Experimental results on two datasets from an app search engine demonstrate that our model can make effective use of semantic knowledge, and thus can significantly enhance the accuracies of latent matching models, particularly for tail queries.
BibTeX
@inproceedings{wang2016incorporating,
  author = {Shuxin Wang and Xin Jiang and Hang Li and Jun Xu and Bin Wang},
  title = {{Incorporating Semantic Knowledge into Latent Matching Model in Search}},
  booktitle = {AIRS},
  year = {2016},
  doi = {10.1007/978-3-319-48051-0\_3},
}
Figure