On the Robustness of Question Rewriting Systems to Questions of Varying Hardness

Hai Ye, Hwee Tou Ng, Wenjuan Han

Introduction

In Conversational Question Answering (CQA), user queries are often context-dependent, containing ellipses or anaphora (e.g., "When was he born?"). Question Rewriting (QR) systems aim to transform these into standalone, self-contained questions (e.g., "When was Corbin Bleu born?") to facilitate downstream QA tasks. However, existing QR systems often struggle with complex rewrites while performing well on simpler ones. This paper investigates the robustness of QR systems across varying levels of question hardness and proposes a framework to enhance performance through difficulty-aware specialization.

Problem Formulation: Defining Hardness

The authors identify that a primary challenge in QR is the lack of a standardized metric for task difficulty. They propose a discrepancy-based heuristic to quantify hardness, defined as the similarity between the original question ($q$) and its gold-standard rewrite ($q'$). Using the BLEU score as a proxy for similarity, they categorize questions into three difficulty levels:

  • Easy: High BLEU score (minimal changes, typically simple pronoun replacement).
  • Medium: Moderate BLEU score.
  • Hard: Low BLEU score (significant changes, requiring synthesis of information from multiple turns or expansion of vague queries).

Human analysis confirms that "easy" questions almost exclusively involve pronoun replacement, while "hard" questions require complex linguistic operations, such as adding prepositional phrases or clarifying "else" contexts, which standard models frequently fail to capture.

Proposed Methodology

To address the performance gap on complex questions, the paper introduces a difficulty-aware architecture based on BART-base.

Difficulty-Aware Adapters

The architecture leverages a BART-base model augmented with Adapters—compact, trainable modules integrated into the transformer layers. Instead of fine-tuning a single large model on all data, the authors train "private" adapters specifically for each difficulty subset (Hard, Medium, and Easy). This allows the model to learn specialized rewriting strategies for different levels of complexity.

Model Ensemble Strategies

To handle unknown difficulty at inference time, two strategies are proposed to combine the specialized adapters:

  1. Sequence-level Adapter Fusion (SLAF): A difficulty classifier predicts the hardness of the input question. The resulting class distribution is used to compute a weighted fusion of the outputs from the specialized adapters.
  2. Sequence-level Adapter Distillation (SLAD): To mitigate the computational overhead of running multiple adapters, the knowledge from the specialized "teacher" models is distilled into a single "student" model.
Overview of the proposed difficulty-aware architecture, including Adapter Fusion and Distillation strategies.

Experimental Setup

The framework is evaluated on two benchmark datasets:

  • CANARD: Derived from QuAC, featuring human-rewritten standalone questions.
  • QRECC: A large-scale dataset with over 80,000 question-answer pairs across diverse conversational contexts.

Results and Discussion

The proposed specialized models consistently outperform standard baselines across various metrics.

Performance Gains

On the CANARD dataset, the SLAD model achieved a mean BLEU score of 53.35, surpassing both the standard BART baseline (53.14) and traditional LSTM-based models (52.16). The most significant improvements were observed in the "Hard" subset. On QRECC, the SLAF model improved the BLEU score for hard questions by nearly 2 points compared to a shared baseline.

Insights on Specialization

The results demonstrate that training on specific difficulty subsets allows models to better capture the nuances of complex rewrites. While simple "copy mechanisms" in LSTMs are sufficient for easy pronoun swaps, they fail on hard questions that require generating context-heavy phrases. Pre-trained models like BART provide a stronger foundation but still benefit significantly from difficulty-specific tuning.

Oracle Upper Bound

An "Oracle" experiment (Mix-gold), which assumes perfect knowledge of question difficulty, reached a mean BLEU of 56.55 on CANARD. This indicates that the primary bottleneck for current QR systems is the accuracy of the difficulty classifier.

Conclusion

This research highlights the importance of accounting for task difficulty in Question Rewriting. By utilizing specialized adapters and ensemble techniques like fusion and distillation, the proposed framework achieves state-of-the-art results, particularly on complex, "hard" questions. Future work may explore optimizing model architectures for each difficulty level to further push the boundaries of robust conversational understanding.

Summary updated

Abstract

In conversational question answering (CQA), the task of question rewriting~(QR) in context aims to rewrite a context-dependent question into an equivalent self-contained question that gives the same answer. In this paper, we are interested in the robustness of a QR system to questions varying in rewriting hardness or difficulty. Since there is a lack of questions classified based on their rewriting hardness, we first propose a heuristic method to automatically classify questions into subsets of varying hardness, by measuring the discrepancy between a question and its rewrite. To find out what makes questions hard or easy for rewriting, we then conduct a human evaluation to annotate the rewriting hardness of questions. Finally, to enhance the robustness of QR systems to questions of varying hardness, we propose a novel learning framework for QR that first trains a QR model independently on each subset of questions of a certain level of hardness, then combines these QR models as one joint model for inference. Experimental results on two datasets show that our framework improves the overall performance compared to the baselines.
BibTeX
@inproceedings{ye2022robustness,
  author = {Hai Ye and Hwee Tou Ng and Wenjuan Han},
  title = {{On the Robustness of Question Rewriting Systems to Questions of Varying Hardness}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=b_8jxlo0o_},
}
Figure