Sequence to Sequence Learning for Query Expansion

Salah Zaiem, Fatiha SadatAAAI 2019 · DOI

TL;DR

This paper explores using sequence-to-sequence (Seq2Seq) neural networks for query expansion, a task traditionally tackled with relevance feedback or word embeddings. While achieving modest improvements, the authors highlight the challenges of capturing nuanced semantic relationships and suggest future work involving search-integrated training for more robust query expansion.

What Problem Does This Paper Address?

In information retrieval, query expansion is a crucial technique to improve search results by adding relevant terms to an initial query. Traditional methods like Rocchio's algorithm rely on document relevance feedback, while more recent approaches leverage word embeddings to find semantically similar terms. However, the authors note that using advanced neural network architectures like Seq2Seq for this task, particularly in a document-independent manner, has been largely unexplored. They aim to bridge this gap by investigating whether Seq2Seq models can effectively learn to expand queries by capturing complex word relationships, even without direct access to the documents being searched.

Methods

The core of the paper is the application of a Seq2Seq neural network for query expansion. This architecture, commonly used in machine translation, consists of an encoder that converts an input sequence (the original query) into a fixed-length vector representation, and a decoder that generates an output sequence (the expanded query) from this vector.

Key methodological aspects include:

  • Document-Independent Expansion: The system is designed to expand queries without needing to analyze the documents themselves.
  • Keyword Extraction for Training Data: To build their training set, the authors employed sentence embeddings to identify keywords that contribute most to a sentence's meaning. This was done by analyzing the hidden states of an encoder (specifically, the Infersent model).
  • Training Data Construction: They utilized existing datasets like MultiNLI, SNLI, and MSCOCO. For Quora question pairs, they used words not present in the first formulation as potential expansions. MSCOCO captions, describing the same image, were used to infer expansion terms by comparing words across different descriptions.
  • Model Architecture: A bidirectional LSTM encoder and a 2-layer LSTM decoder were used, initialized with pre-trained GloVe word embeddings.
  • Training Details: The model was trained using Stochastic Gradient Descent with a learning rate of 0.001 and a decay of 0.5 per epoch, for 25 epochs.

Datasets

The authors used several publicly available datasets for training and evaluation:

  • Training Data:
    • MultiNLI (Multi-Genre Natural Language Inference): Used for its sentence pairs, with contradictory classifications removed.
  • SNLI (Stanford Natural Language Inference): Similar to MultiNLI, used for sentence pairs, excluding contradictions.
  • Quora Question Pairs: Used to extract words not present in the initial formulation of a question as potential expansions.
  • MSCOCO (Microsoft Common Objects in Context): A large dataset of image captions. Words appearing in one caption but not another for the same image were treated as potential expansions.
  • Evaluation Data:
    • TREC Robust 2004 Dataset: Used for Information Retrieval evaluation, consisting of 250 queries and 528,155 documents.
  • WikiQA Dataset: Used for Answer Preselection evaluation.

Key Insights and Ideas

  • Seq2Seq for Query Expansion is Novel: The paper pioneers the application of Seq2Seq models for query expansion, moving beyond traditional methods.
  • Document-Independent Expansion is Feasible: The approach demonstrates that query expansion can be performed without direct document access, relying solely on learned semantic relationships from training data.
  • Challenges in Capturing Deep Semantics: The authors acknowledge that their QE system struggles to capture complex semantic mechanisms, leading to a significant portion of queries not being expanded. This suggests that current embedding spaces might not fully encapsulate all necessary semantic relationships for robust expansion, especially for unseen topics.
  • The Importance of Training Data Quality: The method for constructing training data from existing corpora (like MultiNLI, SNLI, MSCOCO) is a key innovation, enabling document-independent learning.

Key Results

The paper reports modest improvements in performance across two tasks:

  • Information Retrieval (using TREC Robust 2004):
    • Using TF-IDF weighting, the Mean Average Precision (MAP) increased from 0.2517 (without QE) to 0.2581 (with QE).
  • Using BM25 weighting, MAP increased from 0.2709 (without QE) to 0.2783 (with QE).
  • Answer Preselection (using WikiQA):
    • Accuracy improved slightly from 0.2871 to 0.2889.
  • Coverage improved slightly from 0.7840 to 0.7901.

Qualitatively, the authors note that the improvements are "far from being impressive" and not statistically significant, indicating room for substantial enhancement.

Limitations and Future Work

The authors explicitly identify several limitations and propose future directions:

  • Limited Semantic Understanding: The QE system fails to capture nuanced semantic mechanisms, particularly for unseen topics. This results in 45% of queries not being expanded at all.
  • Reliance on Examples: The model learns expansions from examples, struggling to generalize to novel concepts or relationships not present in the training data.
  • Future Work - Search-Integrated Training: A promising avenue is to incorporate the search process directly into the training loop. This would allow the model to learn from search results, using search performance as a reward signal to refine the encoder-decoder network. This "search-oriented" training is expected to yield better precision and accuracy.
  • Future Work - Local Entailments: Training the model on local entailment tasks (rather than broad-coverage inference) might lead to more effective expansion for specific topics.
  • Future Work - More Training Data: Increasing the volume and diversity of training data is expected to improve performance.

Summary updated

BibTeX
@inproceedings{zaiem2019sequence,
  author = {Salah Zaiem and Fatiha Sadat},
  title = {{Sequence to Sequence Learning for Query Expansion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.330110075},
}
Figure