Unsupervised FAQ Retrieval with Question Generation and BERT

Yosi Mass, Boaz Carmeli, Haggai Roitman, David KonopnickiACL 2020 · OpenReview

Introduction

Frequently Asked Questions (FAQ) retrieval is a critical component of automated customer support, aiming to match user natural language queries to the most relevant question-answer pairs. Traditionally, this task is approached via two primary strategies: Q-to-a matching (query to answer text) and Q-to-q matching (query to question text). While Q-to-q matching is generally more effective due to structural similarities between queries and FAQ questions, training high-performing models typically requires extensive query logs—human-annotated datasets mapping real user queries to FAQ pairs. In many real-world applications, such labeled data is unavailable, forcing a reliance on less accurate keyword-matching techniques.

Researchers at IBM Research AI have addressed this "labeled data gap" by developing a fully unsupervised FAQ retrieval method. By leveraging GPT-2 for synthetic question generation and fine-tuning BERT models through weak supervision, their approach rivals and occasionally surpasses supervised models without requiring manual annotation.

Methodology

The proposed system utilizes a multi-stage pipeline that transitions from broad retrieval to precise ranking using three unsupervised "experts":

  1. Initial Retrieval: A standard BM25 search (via Elasticsearch) is performed against both the FAQ question and answer fields to generate a top-k candidate pool.
  2. Passage-based Re-ranking (bm25-maxpsg): This expert employs a sliding window over the answer text, scoring candidates based on the single best-matching passage rather than the entire document.
  3. BERT for Q-to-a Similarity: A BERT model is fine-tuned to recognize the relationship between questions and their corresponding answers. Since FAQ pairs are inherently linked, this provides "free" weak supervision.
  4. BERT for Q-to-q Similarity (Generative Paraphrasing): To enable Q-to-q matching without real queries, the authors use Generative Question Generation. GPT-2 is fine-tuned on existing FAQ pairs to learn how to generate questions from answers. It then produces multiple paraphrases for every FAQ question. To ensure data quality, a "cycle-consistency" filter is applied: only paraphrases that can successfully retrieve the original answer via search are retained. A BERT model is then trained on these high-quality synthetic (paraphrase, original question) pairs.

The final scores are aggregated using unsupervised fusion techniques such as CombSUM or PoolRank, the latter of which incorporates a query expansion step.

Evaluation

The framework was evaluated on two benchmark datasets:

  • FAQIR: A dataset from the Yahoo! Answers "maintenance & repair" domain, featuring 4,313 FAQ pairs and 1,233 user queries.
  • StackFAQ: Derived from Stack Exchange's "web apps" section, containing 719 FAQ pairs and 1,249 user queries.

Key Results

The unsupervised approach demonstrated remarkable efficacy across both benchmarks. On the FAQIR dataset, the PoolRank method achieved a Mean Reciprocal Rank (MRR) of 0.88, outperforming supervised baselines such as CNN-Rank (0.85) and LambdaMART (0.84). On StackFAQ, the model reached an MRR of 0.88 (and 0.90 with PoolRank+), significantly narrowing the gap with the top supervised baseline, TSUBAKI+BERT (0.94).

Table 1: Performance comparison of unsupervised and supervised retrieval methods on FAQIR and StackFAQ datasets.

Analysis of individual components revealed that the BERT-Q-q model, trained on synthetic paraphrases, was the strongest individual re-ranker. This confirms that generative models like GPT-2 can effectively "hallucinate" user intent to bridge the lack of real-world query logs. Furthermore, the results highlight that late fusion—combining lexical (BM25) and semantic (BERT) signals—consistently yields superior performance.

Conclusion and Future Work

This research demonstrates that synthetic data can serve as a robust bridge for low-resource retrieval tasks. The "cycle-consistency" check proved vital for maintaining precision when training on generated text. While highly effective, the authors note that computational overhead and domain-specific generation quality (e.g., in medical or legal fields) remain areas for further exploration. Future work may extend this framework to other low-resource tasks where structured pairs, such as document-summary or title-body, are available but labeled queries are not.

Summary updated

BibTeX
@inproceedings{mass2020unsupervised,
  author = {Yosi Mass and Boaz Carmeli and Haggai Roitman and David Konopnicki},
  title = {{Unsupervised FAQ Retrieval with Question Generation and BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=UDMNqa7DAY},
}
Figure