Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting

Sarawoot Kongyoung, Craig MacDonald, Iadh OunisEMNLP 2023 · DOI

Abstract

Conversational search systems often struggle with underspecified queries containing pronouns or implicit context. This paper presents a Multi-Task Learning (MTL) framework that jointly performs follow-up identification and query rewriting. By framing both tasks as a single sequence-to-sequence problem using BART and T5 models, the approach achieves superior performance in both classification accuracy and downstream passage retrieval compared to traditional single-task methods.

The Challenge of Conversational Context

In natural dialogue, users frequently omit information or use pronouns that refer back to previous turns. For instance, a user might ask "Was he successful?" after discussing a specific individual. Without the preceding context, a search engine cannot resolve "he" or "successful at what."

Figure 1: An example of conversational question rewriting where pronouns like "he" and "the paper" are resolved using dialogue history.

Effective conversational search requires two distinct but related capabilities: identifying whether a new query is a continuation of the current topic (Follow-up Identification) and transforming that query into a standalone, context-rich version (Question Rewriting). The authors argue that treating these as separate tasks misses the opportunity for the model to develop a shared, deeper understanding of the dialogue.

Proposed Multi-Task Learning Framework

The researchers propose an MTL approach that leverages the shared representation of these two tasks. Instead of treating them as sequential, independent steps, the model is trained to handle both simultaneously.

The core innovation is the Generative MTL setup. In this configuration, the model receives the conversation history and the raw question as input. It then generates a single output string that begins with a classification label (e.g., "follow" or "shift") followed by the rewritten query.

Figure 2: Architectural overview of the MTL Discriminative + Generative (a) and MTL Generative (b) models.

By forcing the model to predict the "follow/shift" intent before generating the rewrite, the internal state is effectively conditioned on the conversational intent, leading to more coherent and contextually accurate results. This "priming" helps the model decide whether to incorporate historical context or treat the new query as a fresh start.

Experimental Setup and Datasets

The models were evaluated using three key datasets:

  • LIF (Learning to Identify Follow-Up): Benchmarking the ability to detect topic continuity.
  • CANARD (Context Abstraction: Necessary Additional Rewritten Discourse): Providing human-annotated rewrites for training the generative component.
  • OR-QuAC: Assessing the impact of rewrites on open-retrieval performance using metrics like Mean Reciprocal Rank (MRR) with BM25 and monoT5 re-ranking.

Key Results and Insights

The MTL approach demonstrated significant improvements across the board:

  • Classification Gains: The Generative MTL BART model improved F1 scores by +3.5% to +10.5% over previous state-of-the-art attentive pooling networks.
  • Retrieval Excellence: On the OR-QuAC benchmark, the MTL T5 model achieved an MRR of 0.2150, a substantial jump from the 0.1843 achieved by the standard T5 baseline.
  • Entity Resolution: Qualitative analysis indicated that joint learning helps the model distinguish between similar entities more effectively. For example, the MTL model was better at identifying specific names (like "Disco Volante") by correctly leveraging the dialogue history where single-task models failed.

Conclusion and Future Work

Treating follow-up identification and question rewriting as a unified task allows Transformer-based models to develop a deeper understanding of conversational dynamics. The MTL framework not only simplifies the pipeline by using a single model for multiple tasks but also delivers measurable gains in search relevance and query clarity.

While successful, the authors note that future iterations could benefit from real-time user feedback loops and the ability to generate clarification queries when a user's intent remains ambiguous despite the available context.

Summary updated

BibTeX
@inproceedings{kongyoung2023multi,
  author = {Sarawoot Kongyoung and Craig MacDonald and Iadh Ounis},
  title = {{Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.913},
}
Figure