PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking

Matthew Henderson, Ivan Vulic, Iñigo Casanueva, Pawel Budzianowski, Daniela Gerz, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrksic, Pei-Hao Su

Abstract

We present PolyResponse, a conversational search engine that supports task-oriented dialogue. It is a retrieval-based approach that bypasses the complex multi-component design of traditional task-oriented dialogue systems and the use of explicit semantics in the form of task-specific ontologies. The PolyResponse engine is trained on hundreds of millions of examples extracted from real conversations: it learns what responses are appropriate in different conversational contexts. It then ranks a large index of text and visual responses according to their similarity to the given context, and narrows down the list of relevant entities during the multi-turn conversation. We introduce a restaurant search and booking system powered by the PolyResponse engine, currently available in 8 different languages.

Introduction

Traditional task-oriented dialogue systems (like those used for booking flights or restaurants) are typically built as a pipeline of specialized components: Natural Language Understanding (NLU), Dialogue State Tracking (DST), and Natural Language Generation (NLG). This modular approach suffers from rigidity, relying on expensive, manually engineered ontologies (predefined lists of slots like CUISINE or PRICE). Furthermore, training these modules requires massive amounts of human-annotated data, which is rare and domain-specific.

PolyResponse addresses these challenges by treating dialogue as a ranking problem rather than a generation or state-tracking problem. By training on hundreds of millions of real-world conversations, it learns to "find" the best text or visual response for a given context, bypassing the need for brittle, manually engineered domain ontologies.

Methods: A Unified Ranking Architecture

The core of PolyResponse is a multi-modal encoder architecture that maps conversational contexts and potential responses into a shared 512-dimensional vector space.

The PolyResponse architecture uses dual and triple encoders to map text and images into a shared vector space for ranking.

The system employs three primary encoders:

  • Context Encoder ($f$): Processes the conversational history into a vector $h_c$.
  • Response Encoder ($g$): Maps candidate text replies (e.g., from Reddit or Yelp reviews) into $h_r$.
  • Photo Encoder ($t$): Uses a pre-trained CNN (MobileNet) to map images into $h_p$.

The relevance of a candidate response $r$ to a context $c$ is determined by a scaled cosine similarity score: $S(r, c) = C \cdot \cos(h_r, h_c)$. This retrieval-based approach ensures that responses are natural, grounded in real-world data, and free from the "hallucinations" common in generative models.

Datasets and Training

To avoid the "small data" trap of traditional dialogue research, the authors leverage massive, publicly available datasets:

  • Reddit: A corpus of 727 million filtered context-reply pairs used to train the general conversational encoders.
  • Yelp Open Dataset: Approximately 200,000 photos and captions used to train the multi-modal ranking component.
  • Local Entity Data: For specific applications like restaurant search, the system indexes thousands of photos and sentences extracted from online reviews to serve as candidate responses.

Key Results and Multilingual Support

PolyResponse demonstrates significant scalability and flexibility, supporting 8 different languages across various cities. The system utilizes a machine-translation-based approach to map non-English queries into the English embedding space, allowing for zero-shot cross-lingual retrieval.

PolyResponse deployed in multiple languages (English, French, German, and Chinese) across different cities.

Key insights from the deployment include:

  • Implicit Semantics: The model doesn't need explicit slot definitions. It implicitly understands that "Is it expensive?" and "How much for a meal?" should map to similar vector spaces based on its large-scale training.
  • Multi-modality: Because the system is based on vector similarity, adding images is mathematically identical to adding text, allowing for a seamless "voice-in, visual-out" user experience.
  • Efficiency: By using libraries like Faiss for nearest-neighbor search, the system scales to billions of candidate responses while maintaining real-time performance.

Limitations and Future Work

While the retrieval-based approach excels at search and discovery, the "final mile" of task completion—such as entering specific dates or party sizes for a booking—still switches to a traditional slot-filling flow. Future work aims to replace the translation-dependent multilingual support with native multilingual encoders and to develop more sophisticated, learned policies for "entity convergence" during multi-turn conversations.

Summary updated

Abstract

We present PolyResponse, a conversational search engine that supports task-oriented dialogue. It is a retrieval-based approach that bypasses the complex multi-component design of traditional task-oriented dialogue systems and the use of explicit semantics in the form of task-specific ontologies. The PolyResponse engine is trained on hundreds of millions of examples extracted from real conversations: it learns what responses are appropriate in different conversational contexts. It then ranks a large index of text and visual responses according to their similarity to the given context, and narrows down the list of relevant entities during the multi-turn conversation. We introduce a restaurant search and booking system powered by the PolyResponse engine, currently available in 8 different languages.
BibTeX
@inproceedings{henderson2019polyresponse,
  author = {Matthew Henderson and Ivan Vulic and Iñigo Casanueva and Pawel Budzianowski and Daniela Gerz and Sam Coope and Georgios Spithourakis and Tsung-Hsien Wen and Nikola Mrksic and Pei-Hao Su},
  title = {{PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=s5GNuYJdMZG},
}
Figure